Skip to contentSkip to conversation

sysfs per-cpu cpufreq subdirs/symlinks screwed up after s2ram

Bug 55411 Viresh Kumar 5 participants 38 messages Mar 19, 2013 – Mar 29, 2013 2 messages not in the archive CPU frequency scaling framework

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

A message that is not in the archive (sent off-list or not yet received): [email protected]

Viresh Kumar

author maintainer Tested-by Shawn Guo
#2 lore raw

Hi Guys,

We are talking here about a bug reported by Duncan here. His cpu/cpu*/cpufreq directory are getting corrupted with 3.9-rc3 and was working well with 3.8

https://bugzilla.kernel.org/show_bug.cgi?id=55411

On his AMD bulldozer tri-cluster/6-core system he doesn't see affected and related
cpus set correctly after off-lining 1-5 and bringing them back with:

for i in 1 2 3 4 5; do echo 0 > /sys/devices/system/cpu/cpu$i/online ; done for i in 1 2 3 4 5; do echo 1 > /sys/devices/system/cpu/cpu$i/online ; done

Before running above two, cpufreq-info gave:
https://bugzilla.kernel.org/attachment.cgi?id=95701

And after running above it gave:
https://bugzilla.kernel.org/attachment.cgi?id=95711

Clearly it got corrupted. Somehow cpu 3 showed up in related cpus field of cpu 5.

I suspect following patches behind this:

commit fcf8058296edbc3de43adf095824fc32b067b9f8

Author: Viresh Kumar <[email protected]>
Date:   Tue Jan 29 14:39:08 2013 +0000

cpufreq: Simplify cpufreq_add_dev()

Currently cpufreq_add_dev() firsts allocates policy, calls driver->init() and then checks if this CPU is already managed or not. And if it is already managed, its policy is freed.

We can save all this if we somehow know that CPU is managed or not in advance. policy->related_cpus contains the list of all valid sibling CPUs of policy->cpu. We can check this to see if the current CPU is already managed.

From now on, platforms don't really need to set related_cpus from their init() routines, as the same work is done by core too.

If a platform driver needs to set the related_cpus mask with some additional CPUs, other than CPUs present in policy->cpus, they are free to do it, though, as we don't override anything.

AND

commit 643ae6e81dd65b333a13259852405fc9f764ac76

Author: Viresh Kumar <[email protected]>
Date:   Sat Jan 12 05:14:38 2013 +0000

cpufreq: Manage only online cpus

cpufreq core doesn't manage offline cpus and if driver->init() has returned mask including offline cpus, it may result in unwanted behavior by
cpufreq core
or governors.

We need to get only online cpus in this mask. There are two places
to fix this
mask, cpufreq core and cpufreq driver. It makes sense to do this
at common place
and hence is done in core.

And this is the latest piece of documentation available:

SMP systems normally have same clock source for a group of cpus. For these the .init() would be called only once for the first online cpu. Here the .init() routine must initialize policy->cpus with mask of all possible cpus (Online + Offline) that share the clock. Then the core would copy this mask onto policy->related_cpus and will reset policy->cpus to carry only online cpus.

I saw acpi-cpufreq drivers driver->init() code and found it is not yet aligned to this
theory and probably that is causing these failures.

I don't have enough knowledge about this driver and how is it used for all x86 systems and so want somebody else (who has some prior experience with it) to check how policy->cpus and policy->related_cpus must be set from driver->init().

--
viresh

---------- Forwarded message ----------
From: <[email protected]>
Date: 19 March 2013 13:19
Subject: [Bug 55411] sysfs per-cpu cpufreq subdirs/symlinks screwed up after s2ram
To: [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=55411

--- Comment #9 from Duncan <[email protected]> 2013-03-19 07:49:53 --- (In reply to comment #8)

Quoted 8 lines

(In reply to comment #0)

Quoted 6 lines

After a s2ram/resume cycle (now bad):

/sys/devices/system/cpu/cpu0/cpufreq/ /sys/devices/system/cpu/cpu1/cpufreq -> ../cpu0/cpufreq/ /sys/devices/system/cpu/cpu3/cpufreq/ /sys/devices/system/cpu/cpu5/cpufreq/

Quoted 8 lines

Can you try this rather than s2r:

for i in 1 2 3 4 5; do echo 0 > /sys/devices/system/cpu/cpu$i/online ; done for i in 1 2 3 4 5; do echo 1 > /sys/devices/system/cpu/cpu$i/online ; done

and check the status if things are still corrupted for you?

Above doesn't corrupt anything for me Atleast.

That's a nice easy test; no rebuild and reboot needed. =:^)

Tho I had to change the > to >| as I have bash noclobber set and the files obviously already exist...

Uncorrupted before the test, corrupted after. So just cycling the cpus off and then back online *DOES* corrupt cpufreq, thus a much simpler reproducer! =:^) Exact same ls results as the above.

Quoted 1 line

And my system doesn't have S2R support for now.

My old system didn't support s2ram reliably; it would work occasionally but mostly it didn't. But s2disk was workable for awhile, until the fact that I was running mdraid and the disks didn't always return in the same sdX slots due to hardware wakeup issues complicated things, so eventually I didn't use that much either. The new system's great with s2ram, sans this bug of course; s2disk didn't work at all at first, but last time I tried it /almost/ worked so there has been improvement. But I don't like to take unnecessary chances with filesystem log replay and thankfully wall power's good enough around here that I can s2ram for a day and come back and wiggle the mouse and all's fine (with a couple pre-suspend syncs thrown into my script just in case), so I tend to use it a LOT, even more than I'd use s2disk due to the speed. =:^)

But I'd love to have s2both working reliably; for all I know it's actually working now; it was pretty close. But I prefer not to test the reiserfs log replay (even with pre-suspend syncs I worry, tho as I said reiserfs has actually been very good to me even thru faulty ram, a power supply blowing up on me, a mobo dying, etc, since 2.6.16 or whenever it was that it got ordered journaling by default) when it doesn't work, so knowing s2disk didn't work well when I tested it and with s2ram working SO well, I don't tend to test s2disk/s2both too often.

Meanwhile, thanks for the cpuinfo_cur_freq explanation. If that actually real-time touches the hardware to get the data as you say, that does explain the root privs. Maybe that bit of extra info could be added to the documentation? I could propose some new wording and open a new bug on cpu-freq/user-guide.txt for it if appropriate.

--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

#3 lore raw

[Adding Boris and Thomas to the CC.]

Quoted 85 lines · Viresh Kumar

Hi Guys,

We are talking here about a bug reported by Duncan here. His cpu/cpu*/cpufreq directory are getting corrupted with 3.9-rc3 and was working well with 3.8

https://bugzilla.kernel.org/show_bug.cgi?id=55411

On his AMD bulldozer tri-cluster/6-core system he doesn't see affected and related
cpus set correctly after off-lining 1-5 and bringing them back with:

for i in 1 2 3 4 5; do echo 0 > /sys/devices/system/cpu/cpu$i/online ; done for i in 1 2 3 4 5; do echo 1 > /sys/devices/system/cpu/cpu$i/online ; done

Before running above two, cpufreq-info gave:
https://bugzilla.kernel.org/attachment.cgi?id=95701

And after running above it gave:
https://bugzilla.kernel.org/attachment.cgi?id=95711

Clearly it got corrupted. Somehow cpu 3 showed up in related cpus field of cpu 5.

I suspect following patches behind this:

commit fcf8058296edbc3de43adf095824fc32b067b9f8

Author: Viresh Kumar <[email protected]>
Date:   Tue Jan 29 14:39:08 2013 +0000

cpufreq: Simplify cpufreq_add_dev()

Currently cpufreq_add_dev() firsts allocates policy, calls driver->init() and then checks if this CPU is already managed or not. And if it is already managed, its policy is freed.

We can save all this if we somehow know that CPU is managed or not in advance. policy->related_cpus contains the list of all valid sibling CPUs of policy->cpu. We can check this to see if the current CPU is already managed.

From now on, platforms don't really need to set related_cpus from their init() routines, as the same work is done by core too.

If a platform driver needs to set the related_cpus mask with some additional CPUs, other than CPUs present in policy->cpus, they are free to do it, though, as we don't override anything.

AND

commit 643ae6e81dd65b333a13259852405fc9f764ac76

Author: Viresh Kumar <[email protected]>
Date:   Sat Jan 12 05:14:38 2013 +0000

cpufreq: Manage only online cpus

cpufreq core doesn't manage offline cpus and if driver->init() has returned mask including offline cpus, it may result in unwanted behavior by
cpufreq core
or governors.

We need to get only online cpus in this mask. There are two places
to fix this
mask, cpufreq core and cpufreq driver. It makes sense to do this
at common place
and hence is done in core.

And this is the latest piece of documentation available:

SMP systems normally have same clock source for a group of cpus. For these the .init() would be called only once for the first online cpu. Here the .init() routine must initialize policy->cpus with mask of all possible cpus (Online + Offline) that share the clock. Then the core would copy this mask onto policy->related_cpus and will reset policy->cpus to carry only online cpus.

Quoted 8 lines

I saw acpi-cpufreq drivers driver->init() code and found it is not yet aligned to this
theory and probably that is causing these failures.

I don't have enough knowledge about this driver and how is it used for all x86 systems and so want somebody else (who has some prior experience with it) to check how policy->cpus and policy->related_cpus must be set from driver->init().

OK, so what exactly do you need to now?

This has to be addressed before final 3.9 this way or another - and the sooner the better.

Thanks,
Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
author maintainer
#4 lore raw
Quoted 1 line · Rafael J. Wysocki

OK, so what exactly do you need to now?

s/now/know ??

We just need to set policy->cpus with all cpus (online or offline) that share clock line with cpu for which init() is called.

cpufreq core with then set all those cpus onto related cpus and will keep only online cpus in policy->cpus..

Sorry if i haven't answered it well :(

#5 lore raw
Quoted 4 lines · Viresh Kumar
Quoted 1 line · Rafael J. Wysocki

OK, so what exactly do you need to now?

s/now/know ??

Yes, sorry.

#6 lore raw
Quoted 7 lines · Rafael J. Wysocki
Quoted 4 lines · Viresh Kumar
Quoted 1 line · Rafael J. Wysocki

OK, so what exactly do you need to now?

s/now/know ??

Yes, sorry.

Right, I heard about this breakage on Bulldozer and I can repro it here too :(.

Which is not good and it needs to be fixed ASAP. From looking at the patches in question, this should be reproducible on everything, though, since it is generic cpufreq code. And if so, this makes the urgency of a fix much more dire.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#7 lore raw
In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

[Adding Boris and Thomas to the CC.]

...

Quoted 9 lines
Quoted 4 lines

I don't have enough knowledge about this driver and how is it used for all x86 systems and so want somebody else (who has some prior experience with it) to check how policy->cpus and policy->related_cpus must be set from driver->init().

OK, so what exactly do you need to now?

This has to be addressed before final 3.9 this way or another - and the sooner the better.

Is this all to try to fix "cpufreq driver gets loaded while some cores were set offline before"?

I wonder how you run into "cpufreq is initialized with offlined cpus" case. I remember that there were problems, but it's nearly impossible to run into this if the cpufreq driver is loaded early at boot.

cpufreq_add_dev() and friends are complicated.
Those init functions got split some time ago and there also slipped in a bug even it was simple splitting of functions.

I do not have time to look at fcf8058296edbc3de43adf095824fc32b067b9f8 right now. Don't know how much other stuff depends on it and how sever it is on ARM that cpufreq does not correctly initialize with offlined cpus..., I would revert this patch.

There are other cpu related drivers (at least on x86) which cannot initialize correctly if the CPUs got offlined before driver init. Obviously this is not a clever thing to do.

Thomas

author maintainer
#8 lore raw

Hi guys,

I will answer to questions of both of you in this mail.

Quoted 2 lines · Thomas Renninger

Is this all to try to fix "cpufreq driver gets loaded while some cores were set offline before"?

Not really. There are problems with acpi-cpufreq without that case too.

Quoted 3 lines

I wonder how you run into "cpufreq is initialized with offlined cpus" case. I remember that there were problems, but it's nearly impossible to run into this if the cpufreq driver is loaded early at boot.

I always thought there is a way not to boot all cpus by passing stuff in command line and so this is a easy case to reproduce.

Quoted 1 line

cpufreq_add_dev() and friends are complicated.

Not anymore, they are hugely simplified now.

Quoted 7 lines

Those init functions got split some time ago and there also slipped in a bug even it was simple splitting of functions.

I do not have time to look at fcf8058296edbc3de43adf095824fc32b067b9f8 right now. Don't know how much other stuff depends on it and how sever it is on ARM that cpufreq does not correctly initialize with offlined cpus..., I would revert this patch.

Let me clear it to everybody. There isn't/shouldn't be a bug in cpufreq core, its just that acpi-cpufreq driver isn't adapted well with the changes related to affected_cpus and related_cpus.

I have never gone into coding for any non-ARM platform and am really not aware of acpi-cpufreq driver and its users. That's why i told everybody where the issue is, and they just need to fix acpi-cpufreq driver with right values of policy->cpus (affected_cpus) and everything else would work after that.

--
viresh

#9 lore raw
Quoted 5 lines · Viresh Kumar

I have never gone into coding for any non-ARM platform and am really not aware of acpi-cpufreq driver and its users. That's why i told everybody where the issue is, and they just need to fix acpi-cpufreq driver with right values of policy->cpus (affected_cpus) and everything else would work after that.

No, you're breaking existing drivers with your changes - nobody will fix stuff *you're* breaking.

You need to get the hardware and test your changes on *that* hardware too. Otherwise, the offending patches should be reverted until you can provide a patchset which works on everything.

Geez, the fact that I even need to explain this...

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#10 lore raw
Quoted 8 lines · Borislav Petkov

No, you're breaking existing drivers with your changes - nobody will fix stuff *you're* breaking.

You need to get the hardware and test your changes on *that* hardware too. Otherwise, the offending patches should be reverted until you can provide a patchset which works on everything.

Geez, the fact that I even need to explain this...

You are right in every sense and i can't defend myself on that, no excuse.

When i wrote those patches, i fixed many drivers and wasn't aware that acpi-cpufreq got broken. And only with the mail from Duncan i realized it.

Then i tried to go thorough it & fix it and found there is some stuff in init() which is is sort of per user case dependent. And wasn't sure how exactly to fix it and so shouted for help.

But yes i will give it another try to go through the driver in details and see if i can fix it.

--
viresh

#11 lore raw
Quoted 2 lines · Viresh Kumar

But yes i will give it another try to go through the driver in details and see if i can fix it.

Right, and make sure it is a minimal fix which can go in now. If it appears to get more involved, we'd probably need to revert now and you can retry next merge window.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#12 lore raw
In reply to Viresh Kumar
Quoted 10 lines · Viresh Kumar

Hi guys,

I will answer to questions of both of you in this mail.

Quoted 2 lines · Thomas Renninger

Is this all to try to fix "cpufreq driver gets loaded while some cores were set offline before"?

Not really. There are problems with acpi-cpufreq without that case too.

Quoted 7 lines
Quoted 4 lines

I wonder how you run into "cpufreq is initialized with offlined cpus" case.
I remember that there were problems, but it's nearly impossible to run into this if the cpufreq driver is loaded early at boot.

I always thought there is a way not to boot all cpus by passing stuff in command line and so this is a easy case to reproduce.

I am pretty sure cpuidle states won't initialize and in best case you never get them working on the offlined cpus.
Local APICs won't be set up, ...

Such a parameter will never exist for x86.

Quoted 3 lines
Quoted 1 line

cpufreq_add_dev() and friends are complicated.

Not anymore, they are hugely simplified now.

They were hugely simplified and things are not working anymore and you do not know why...

Quoted 9 lines
Quoted 8 lines

Those init functions got split some time ago and there also slipped in a bug even it was simple splitting of functions.

I do not have time to look at fcf8058296edbc3de43adf095824fc32b067b9f8 right now. Don't know how much other stuff depends on it and how sever it is on ARM that cpufreq does not correctly initialize with offlined
cpus..., I would revert this patch.

Quoted 3 lines

Let me clear it to everybody. There isn't/shouldn't be a bug in cpufreq core, its just that acpi-cpufreq driver isn't adapted well with the changes related to affected_cpus and related_cpus.

And powernow-k8 driver is broken.
The others are not tested that often, I expect they broke as well, right?

Quoted 5 lines

I have never gone into coding for any non-ARM platform and am really not aware of acpi-cpufreq driver and its users. That's why i told everybody where the issue is, and they just need to fix acpi-cpufreq driver with right values of policy->cpus (affected_cpus) and everything else would work after that.

Sorry, I cannot look into this due to lack of time, but I remember that there were reasons why cpufreq_add_dev() was complicated.
Or that it's really easy to mess it up and it's not easy to fix it again.

Thomas

author maintainer
#13 lore raw
Quoted 5 lines · Thomas Renninger

I am pretty sure cpuidle states won't initialize and in best case you never get them working on the offlined cpus.
Local APICs won't be set up, ...

Such a parameter will never exist for x86.

I will see if i can find what i was referring to here.

Quoted 2 lines

They were hugely simplified and things are not working anymore and you do not know why...

I know why, but don't know (for now) how to fix it for acpi-cpufreq.

Quoted 2 lines

And powernow-k8 driver is broken.
The others are not tested that often, I expect they broke as well, right?

acpi-cpufreq is broken and so all others who are using it. Sorry for asking the stupid question now but what's the hierarchy of cpufreq drivers for intel (I will try to go through it now), some drivers use acpi-cpufreq driver?

Quoted 3 lines

Sorry, I cannot look into this due to lack of time, but I remember that there were reasons why cpufreq_add_dev() was complicated.
Or that it's really easy to mess it up and it's not easy to fix it again.

Its not cpufreq_add_dev() that is broken but some changes that were part of the same patch, i.e. part that tried to sort out affected and related cpus.

#14 lore raw
Quoted 4 lines · Viresh Kumar

acpi-cpufreq is broken and so all others who are using it. Sorry for asking the stupid question now but what's the hierarchy of cpufreq drivers for intel (I will try to go through it now), some drivers use acpi-cpufreq driver?

All relevant x86 out there uses acpi-cpufreq. Which means that breaking it, breaks cpufreq on x86. Which is absolutely a no-no.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#15 lore raw
In reply to Thomas Renninger
Quoted 16 lines · Thomas Renninger
Quoted 15 lines · Viresh Kumar

Hi guys,

I will answer to questions of both of you in this mail.

Quoted 2 lines · Thomas Renninger

Is this all to try to fix "cpufreq driver gets loaded while some cores were set offline before"?

Not really. There are problems with acpi-cpufreq without that case too.

Quoted 4 lines

I wonder how you run into "cpufreq is initialized with offlined cpus" case.
I remember that there were problems, but it's nearly impossible to run into this if the cpufreq driver is loaded early at boot.

Quoted 8 lines
Quoted 2 lines

I always thought there is a way not to boot all cpus by passing stuff in command line and so this is a easy case to reproduce.

I am pretty sure cpuidle states won't initialize and in best case you never get them working on the offlined cpus.
Local APICs won't be set up, ...

Such a parameter will never exist for x86.

I take that back.
CPU hot-add (even if CPU is not present at boot time) works. I looked at C-states for that some time ago and it should only work if the hot-add event came in via ACPI events for CPUs which were not initialized at boot time. Better would be to initialize the first time it gets switched online.
Anyway, making such stuff (cpufreq/cpuidle/...) more robust, is certainly a good idea.

...

Quoted 2 lines

And powernow-k8 driver is broken.
The others are not tested that often, I expect they broke as well, right?

And powernow-k8 does not exist anymore..., fortunately I didn't have to look at this stuff for some time.

...

Quoted 1 line

Sorry, I cannot look into this due to lack of time,...

Thomas

author maintainer
#16 lore raw
Quoted 6 lines · Thomas Renninger

I take that back.
CPU hot-add (even if CPU is not present at boot time) works. I looked at C-states for that some time ago and it should only work if the hot-add event came in via ACPI events for CPUs which were not initialized at boot time. Better would be to initialize the first time it gets switched online.

I removed it from my TODO list, thanks :)

Quoted 2 lines

Anyway, making such stuff (cpufreq/cpuidle/...) more robust, is certainly a good idea.

Yes, that was the basic intent of my earlier patches that went into 3.9. It was all about simplifying it and making it robust.

Quoted 2 lines

And powernow-k8 does not exist anymore..., fortunately I didn't have to look at this stuff for some time.

Sure? Driver is still present in mainline.

BTW, i have given an initial fix for acpi-cpufreq (which should work) and waiting
for Duncan to reply back. All other drivers don't set affected/related cpus directly,
so they should be alright.

--
viresh

author maintainer
#17 lore raw

Hi Duncan,

Please reply to this mail rather than using bugzilla as all others might not be following bugzilla rerport.

Quoted 9 lines

--- Comment #31 from Duncan <[email protected]> 2013-03-24 09:48:41 --- (In reply to comment #28)

Quoted 2 lines

My weekend is already spoiled (due to my bugs :) ) and i don't want to spoil yours.

Don't worry about it. They're all days in the week, to me. I don't really have a defined "weekend". In fact, I had been (somewhat impatiently) grumbling to myself Friday that I was likely to have to wait until Monday for something concrete to test, so I'm happy to be demonstrated wrong! =:^)

:)

Quoted 6 lines
Quoted 1 line

[PATCH] cpufreq: acpi-cpufreq: Set policy->cpus correctly from .init()

You appear to be on the right path as all the dirs and symlinks are there now, but it looks like you'll need a v2 as the order/pairing is now very strange, both as booted and after a s2ram/resume (which changes the order but it's still strange):

:(

Quoted 6 lines

Testing against 3.9-rc4 now.

Patch applied pre-suspend ls -dl as above (original pairing is 0/1, 2/3, 4/5, with the first always a dir and the second always a symlink to the first, as seen in comment #0):

Quoted 6 lines

/sys/devices/system/cpu/cpu0/cpufreq/ /sys/devices/system/cpu/cpu1/cpufreq -> ../cpu0/cpufreq/ /sys/devices/system/cpu/cpu2/cpufreq/ /sys/devices/system/cpu/cpu3/cpufreq/ /sys/devices/system/cpu/cpu4/cpufreq -> ../cpu3/cpufreq/ /sys/devices/system/cpu/cpu5/cpufreq -> ../cpu2/cpufreq/

The way it works (in cpufreq core), the first cpu of group registered for cpufreq
would get a directory and second one would get a symlink.

And the above ones also doesn't look good. 2/5 and 3/4 shouldn't be the groups.

acpi-cpufreq driver is getting this information from perf->shared_cpu_map and that seems to be wrong to me now.

Quoted 8 lines

Post s2ram/resume cycle:

/sys/devices/system/cpu/cpu0/cpufreq/ /sys/devices/system/cpu/cpu1/cpufreq -> ../cpu0/cpufreq/ /sys/devices/system/cpu/cpu2/cpufreq -> ../cpu4/cpufreq/ /sys/devices/system/cpu/cpu3/cpufreq -> ../cpu5/cpufreq/ /sys/devices/system/cpu/cpu4/cpufreq/ /sys/devices/system/cpu/cpu5/cpufreq/

2/4 and 3/5 are also wrong groups.

Quoted 6 lines

But I don't know whether it's CPU ordering that's weird, or the cpufreq ordering. IOW, I don't know whether it /should/ be 0/1, 2/3, 4/5 or not, because I don't know whether those numbers actually reflect the hardware so it's the ordering above that's bad, or if the cpu numbers are just arbitrarily assigned, and the ordering above reflects the actual hardware, and just looks strange due to the arbitrary cpu numbering.

One thing i am sure about is cpu order is fixed at boot and after boot whatever you do can't change that order... So order should always be 0/1, 2/3, 4/5

Quoted 4 lines

Either way, there's the correct number of dirs and links now, but the ordering is extremely confusing and looks wrong, regardless of whether it actually corresponds to the hardware or not, something I don't have the ability to judge.

Hmm.. Can you try one thing? Run 3.8 over your machine and give output of cpufreq-info and ls -ld after boot and resume..

I would like to see what's the original behavior.

--
viresh

#18 lore raw
Quoted 4 lines · On Sun, 24 Mar 2013 15:32:39 +0530 Viresh Kumar <[email protected]> wrote:

Hi Duncan,

Please reply to this mail rather than using bugzilla as all others might not be following bugzilla rerport.

Thanks. I'm used to it being the other way 'round.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

[PATCH] cpufreq: acpi-cpufreq: Set policy->cpus correctly from .init()

You appear to be on the right path as all the dirs and symlinks are there now, but it looks like you'll need a v2 as the order/pairing is now very strange, both as booted and after a s2ram/resume (which changes the order but it's still strange):

Quoted 4 lines

Hmm.. Can you try one thing? Run 3.8 over your machine and give output of cpufreq-info and ls -ld after boot and resume..

I would like to see what's the original behavior.

Good idea! =:^) It now appears that your bug simply cascaded on a previously unreported bug in earlier kernels.

The 3.8 ls -dl isn't too interesting as they were all dirs not symlinks then, and if they hadn't stuck around after a suspend/resume I'd have definitely noticed and reported the bug back then, but it's still useful to confirm.

The 3.8 pre-suspend and post resume ls -dl are identical -- no missing dirs (and no symlinks):

/sys/devices/system/cpu/cpu0/cpufreq/
/sys/devices/system/cpu/cpu1/cpufreq/
/sys/devices/system/cpu/cpu2/cpufreq/
/sys/devices/system/cpu/cpu3/cpufreq/
/sys/devices/system/cpu/cpu4/cpufreq/
/sys/devices/system/cpu/cpu5/cpufreq/

The interesting results are the 3.8 cpufreq-info. I'll attach the full output, but here's the interesting bit:

3.8 pre-suspend cpufreq-info excerpts (nicely paired, as are the pre-patch pre-suspend results for 3.9-rc):

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0
analyzing CPU 1:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 1
analyzing CPU 2:
CPUs which run at the same hardware frequency: 2 3
CPUs which need to have their frequency coordinated by software: 2
analyzing CPU 3:
CPUs which run at the same hardware frequency: 2 3
CPUs which need to have their frequency coordinated by software: 3
analyzing CPU 4:
CPUs which run at the same hardware frequency: 4 5
CPUs which need to have their frequency coordinated by software: 4
analyzing CPU 5:
CPUs which run at the same hardware frequency: 4 5
CPUs which need to have their frequency coordinated by software: 5

3.8 post-resume (screwed up pairing, so that bit's not a 3.9 thing, I just noticed it in 3.9 due to the now missing symlinks/dirs):

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0
analyzing CPU 1:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 1
analyzing CPU 2:
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
analyzing CPU 3:
CPUs which run at the same hardware frequency: 3
CPUs which need to have their frequency coordinated by software: 3
analyzing CPU 4:
CPUs which run at the same hardware frequency: 2 4
CPUs which need to have their frequency coordinated by software: 4
analyzing CPU 5:
CPUs which run at the same hardware frequency: 3 5
CPUs which need to have their frequency coordinated by software: 5

Signature · 3 lines
Duncan - No HTML messages please, as they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.20%, 2.40 GHz:0.16%, 1.80 GHz:0.20%, 1.40 GHz:99.44% (4)
analyzing CPU 1:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 1
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.03%, 3.00 GHz:0.14%, 2.40 GHz:0.18%, 1.80 GHz:0.35%, 1.40 GHz:99.29% (6)
analyzing CPU 2:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 2 3
CPUs which need to have their frequency coordinated by software: 2
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.15%, 2.40 GHz:0.15%, 1.80 GHz:0.17%, 1.40 GHz:99.54% (4)
analyzing CPU 3:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 2 3
CPUs which need to have their frequency coordinated by software: 3
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.04%, 3.00 GHz:0.11%, 2.40 GHz:0.22%, 1.80 GHz:0.18%, 1.40 GHz:99.45% (4)
analyzing CPU 4:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 4 5
CPUs which need to have their frequency coordinated by software: 4
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.17%, 2.40 GHz:0.30%, 1.80 GHz:0.20%, 1.40 GHz:99.34% (4)
analyzing CPU 5:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 4 5
CPUs which need to have their frequency coordinated by software: 5
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.22%, 2.40 GHz:0.15%, 1.80 GHz:0.30%, 1.40 GHz:99.33% (4)

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.04%, 2.40 GHz:0.03%, 1.80 GHz:0.04%, 1.40 GHz:99.89% (6)
analyzing CPU 1:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 1
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.00%, 2.40 GHz:0.08%, 1.80 GHz:0.04%, 1.40 GHz:99.88% (3)
analyzing CPU 2:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.00%, 2.40 GHz:0.00%, 1.80 GHz:0.76%, 1.40 GHz:99.24% (2)
analyzing CPU 3:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 3
CPUs which need to have their frequency coordinated by software: 3
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.04%, 2.40 GHz:0.04%, 1.80 GHz:0.15%, 1.40 GHz:99.77% (4)
analyzing CPU 4:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 2 4
CPUs which need to have their frequency coordinated by software: 4
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.02%, 3.00 GHz:0.03%, 2.40 GHz:0.02%, 1.80 GHz:0.21%, 1.40 GHz:99.71% (4)
analyzing CPU 5:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 3 5
CPUs which need to have their frequency coordinated by software: 5
maximum transition latency: 4.0 us.
hardware limits: 1.40 GHz - 3.60 GHz
available frequency steps: 3.60 GHz, 3.00 GHz, 2.40 GHz, 1.80 GHz, 1.40 GHz available cpufreq governors: conservative, performance
current policy: frequency should be within 1.40 GHz and 3.60 GHz.
The governor "conservative" may decide which speed to use
within this range.
current CPU frequency is 1.40 GHz.
cpufreq stats: 3.60 GHz:0.00%, 3.00 GHz:0.04%, 2.40 GHz:0.03%, 1.80 GHz:0.12%, 1.40 GHz:99.81% (4)

author maintainer
#19 lore raw
Quoted 9 lines · Duncan
Quoted 4 lines · On Sun, 24 Mar 2013 15:32:39 +0530 Viresh Kumar <[email protected]> wrote:

Hmm.. Can you try one thing? Run 3.8 over your machine and give output of cpufreq-info and ls -ld after boot and resume..

I would like to see what's the original behavior.

Good idea! =:^) It now appears that your bug simply cascaded on a previously unreported bug in earlier kernels.

That made me happy, i am not the only culprit :)

Quoted 9 lines

The 3.8 pre-suspend and post resume ls -dl are identical -- no missing dirs (and no symlinks):

/sys/devices/system/cpu/cpu0/cpufreq/
/sys/devices/system/cpu/cpu1/cpufreq/
/sys/devices/system/cpu/cpu2/cpufreq/
/sys/devices/system/cpu/cpu3/cpufreq/
/sys/devices/system/cpu/cpu4/cpufreq/
/sys/devices/system/cpu/cpu5/cpufreq/

They were all separate directories (instead of symlinks) earlier because this only depended on policy->cpus earlier. And none of the cpus are shared in policy->cpus, i.e. policy->cpus was always policy->cpu.

Quoted 2 lines

3.8 pre-suspend cpufreq-info excerpts (nicely paired, as are the pre-patch pre-suspend results for 3.9-rc):

No they are still not paired well. This is how we should read your analysis:
related-cpus: "same hardware freq"
affected-cpus or policy->cpus: "frequency coordinated by software"

Quoted 2 lines

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0 1

related cpus have correct pairs

Quoted 1 line

CPUs which need to have their frequency coordinated by software: 0

but affected cpus doesn't

Quoted 1 line

3.8 post-resume (screwed up pairing, so that bit's not a 3.9 thing

I told you earlier, this made me happy :)

Quoted 6 lines

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0
analyzing CPU 1:
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 1

These stayed as is as cpu 0 is non removable cpu and so doesn't get unregistered from cpufreq at all.

Quoted 8 lines

analyzing CPU 2:
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
analyzing CPU 3:
CPUs which run at the same hardware frequency: 3
CPUs which need to have their frequency coordinated by software: 3
analyzing CPU 4:
CPUs which run at the same hardware frequency: 2 4

related cpus got corrupted here.

Quoted 4 lines

CPUs which need to have their frequency coordinated by software: 4
analyzing CPU 5:
CPUs which run at the same hardware frequency: 3 5
CPUs which need to have their frequency coordinated by software: 5

Now back to the real issues:

@Rafael/Borislav/Thomas/Andre/Darrick:

"What do we mean by software AND hardware coordination for x86 ?"

Following are the sha-id's which had something to do with above statement.

3b2d99429e3386b6e2ac949fc72486509c8bbe36
46f18e3a28295a9e11a6ffa4478241c19bc93735
acd316248205d553594296f1895ba5196b89ffcc
e8628dd06d66f2e3965ec9742029b401d63434f1
8adcc0c674004c0f9467031a93dc639c2b01411f

On the platform i work (ARM) there are only two cases, cpus share clock line or they don't. So, they share policy struct or they don't.

Fixing Duncan's issues shouldn't be a very big deal now as i was thinking too much about what was broken without my patches too. And now that part is pretty clear.

--
viresh

author maintainer
#20 lore raw
Quoted 3 lines · Viresh Kumar

Fixing Duncan's issues shouldn't be a very big deal now as i was thinking too much about what was broken without my patches too. And now that part is pretty clear.

Hi Duncan,

Try attached patch and this should take back your system to where it was.

NOTE: With this patch your related_cpus wouldn't show any groups and related cpus must be same as affected cpus. All cpu*/cpufreq must be directories now and no symlinks.

--
viresh

From 0eb6a6ce25c71e60dc69e1897f80b7ce3bb41fb2 Mon Sep 17 00:00:00 2001
Message-Id: <0eb6a6ce25c71e60dc69e1897f80b7ce3bb41fb2.1364127667.git.viresh.kumar@linaro.org>
From: Viresh Kumar <[email protected]>
Date: Sat, 23 Mar 2013 23:42:44 +0530
Subject: [PATCH] cpufreq: acpi-cpufreq: Don't set policy->related_cpus from
.init()

With the addition of following patch:

fcf8058 cpufreq: Simplify cpufreq_add_dev()

cpufreq driver's .init() routine must initialize policy->cpus with mask of all possible cpus (Online + Offline) that share the clock. Then the core would copy this mask onto policy->related_cpus and will reset policy->cpus to carry only online cpus.

acpi-cpufreq driver wasn't updated with this assumption and so sometimes when we try to hot[un]plug cpus at run time, sysfs directories gets corrupted.

This patch fixes acpi-cpufreq driver against this corruption.

---
 drivers/cpufreq/acpi-cpufreq.c | 2 --
 1 file changed, 2 deletions(-)
drivers/cpufreq/acpi-cpufreq.c +0 -2
index afbef9c..11b8b4b 100644
@@ -723,7 +723,6 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	    policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) {
 		cpumask_copy(policy->cpus, perf->shared_cpu_map);
 	}
-	cpumask_copy(policy->related_cpus, perf->shared_cpu_map);
 
 #ifdef CONFIG_SMP
 	dmi_check_system(sw_any_bug_dmi_table);
@@ -735,7 +734,6 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	if (check_amd_hwpstate_cpu(cpu) && !acpi_pstate_strict) {
 		cpumask_clear(policy->cpus);
 		cpumask_set_cpu(cpu, policy->cpus);
-		cpumask_copy(policy->related_cpus, cpu_sibling_mask(cpu));
 		policy->shared_type = CPUFREQ_SHARED_TYPE_HW;
 		pr_info_once(PFX "overriding BIOS provided _PSD data\n");
 	}
Signature · 1 line
1.7.12.rc2.18.g61b472e
#21 lore raw
Quoted 5 lines · On Sun, 24 Mar 2013 17:53:41 +0530 Viresh Kumar <[email protected]> wrote:
Quoted 3 lines · Viresh Kumar

Fixing Duncan's issues shouldn't be a very big deal now as i was thinking too much about what was broken without my patches too. And now that part is pretty clear.

Quoted 8 lines

Hi Duncan,

Try attached patch and this should take back your system to where it was.

NOTE: With this patch your related_cpus wouldn't show any groups and related cpus must be same as affected cpus. All cpu*/cpufreq must be directories now and no symlinks.

FWIW, with this patch, pre-s2ram and post-resume are indeed consistent, but it's not back to where it was.

With this patch, each core is a cpufreq law unto itself. Maybe that's what you meant with the note, maybe not (I know the mapping of sysfs files to cpufreq-info output was stated up-thread, but I lost track, and how affected vs related maps to hardware vs software coordinated and what it all actually means other than what I'm seeing isn't ideal, is apparently a bit more than I'm able to keep in my head ATM), but it's what I get. The relevant bits of cpufreq-info output:

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
analyzing CPU 1:
CPUs which run at the same hardware frequency: 1
CPUs which need to have their frequency coordinated by software: 1
analyzing CPU 2:
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
analyzing CPU 3:
CPUs which run at the same hardware frequency: 3
CPUs which need to have their frequency coordinated by software: 3
analyzing CPU 4:
CPUs which run at the same hardware frequency: 4
CPUs which need to have their frequency coordinated by software: 4
analyzing CPU 5:
CPUs which run at the same hardware frequency: 5
CPUs which need to have their frequency coordinated by software: 5

But at least it's consistent: The same results both before and after a suspend/resume cycle.

And given that 3.8 wasn't ideal either, maybe that's good enough for this cycle, and a real fix will have to wait until the next commit window and stable-tree. That'd give us more leeway to fix it right, as well as a full cycle for testing anything else the "correct" fix might dredge up.

Signature · 3 lines
Duncan - No HTML messages please, as they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
author maintainer
#22 lore raw
Quoted 30 lines · Duncan

FWIW, with this patch, pre-s2ram and post-resume are indeed consistent, but it's not back to where it was.

With this patch, each core is a cpufreq law unto itself. Maybe that's what you meant with the note, maybe not (I know the mapping of sysfs files to cpufreq-info output was stated up-thread, but I lost track, and how affected vs related maps to hardware vs software coordinated and what it all actually means other than what I'm seeing isn't ideal, is apparently a bit more than I'm able to keep in my head ATM), but it's what I get. The relevant bits of cpufreq-info output:

analyzing CPU 0:
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
analyzing CPU 1:
CPUs which run at the same hardware frequency: 1
CPUs which need to have their frequency coordinated by software: 1
analyzing CPU 2:
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
analyzing CPU 3:
CPUs which run at the same hardware frequency: 3
CPUs which need to have their frequency coordinated by software: 3
analyzing CPU 4:
CPUs which run at the same hardware frequency: 4
CPUs which need to have their frequency coordinated by software: 4
analyzing CPU 5:
CPUs which run at the same hardware frequency: 5
CPUs which need to have their frequency coordinated by software: 5

Quoted 8 lines

But at least it's consistent: The same results both before and after a suspend/resume cycle.

And given that 3.8 wasn't ideal either, maybe that's good enough for this cycle, and a real fix will have to wait until the next commit window and stable-tree. That'd give us more leeway to fix it right, as well as a full cycle for testing anything else the "correct" fix might dredge up.

This is exactly what i expected and i wrote in Note. Because cpufreq core does a lot of work based on related_cpus now, its better we don't set it blindly for x86.

Following code was the only user of relatead_cpus in 3.8 code:

        /* Set governor before ->init, so that driver could check it */
#ifdef CONFIG_HOTPLUG_CPU
        for_each_online_cpu(sibling) {
                struct cpufreq_policy *cp = per_cpu(cpufreq_cpu_data, sibling);
                if (cp && cp->governor &&
                    (cpumask_test_cpu(cpu, cp->related_cpus))) {
                        policy->governor = cp->governor;
                        found = 1;
                        break;
                }
        }

#endif

There is no other user of relatead_cpus earlier in cpufreq core and that's why i wonder why it was added earlier.

But a grep of relatead_cpus for 3.8 showed some interesting users in:
tools/power/cpupower/

I will try to check what they are doing with it, but for the kernel it was almost unused.
And not it is very much used :)

But for 3.9 i believe this patch should be good enough.

Thanks for testing it.

--
viresh

Tested-by
#23 lore raw
Quoted 1 line · Viresh Kumar

But for 3.9 i believe this patch should be good enough.

Seems so here too.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#24 lore raw
In reply to Viresh Kumar
Quoted 8 lines · Viresh Kumar

There is no other user of relatead_cpus earlier in cpufreq core and that's why i wonder why it was added earlier.

But a grep of relatead_cpus for 3.8 showed some interesting users in:
tools/power/cpupower/

I will try to check what they are doing with it, but for the kernel it was almost unused.

I checked tools and they aren't doing anything tricky with it. Just reading groups of cpus from sysfs.

Now i believe even tools should be patched a bit to give correct meanings of affected or related cpus. Let me try it.

#25 lore raw
In reply to Viresh Kumar
Quoted 4 lines · Viresh Kumar
Quoted 2 lines

And powernow-k8 does not exist anymore..., fortunately I didn't have to look at this stuff for some time.

Sure? Driver is still present in mainline.

powernow-k8 is still there for, well, K8 only. Newer machines are handled by acpi-cpufreq. Thus is acpi-cpufreq's health of major importance especially now. :)

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#26 lore raw

More cpufreq breakage

In reply to Rafael J. Wysocki

Btw,

while we're at it, I just discovered this in dmesg. Plain -rc3.

[34173.893305] ------------[ cut here ]------------
[34173.893321] WARNING: at kernel/mutex.c:199 mutex_lock_nested+0x39c/0x3b0()
[34173.893328] Hardware name: To be filled by O.E.M.
[34173.893333] Modules linked in: nls_iso8859_15 nls_cp437 fuse tun cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative dm_crypt dm_mod ipv6 vfat fat acpi_cpufreq mperf kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 ablk_helper cryptd xts lrw gf128mul microcode radeon amd64_edac_mod edac_core k10temp fam15h_power drm_kms_helper ttm cfbfillrect cfbimgblt r8169 cfbcopyarea
[34173.893395] Pid: 15316, comm: kworker/0:0 Not tainted 3.9.0-rc3 #1
[34173.893402] Call Trace:
[34173.893409]  [<ffffffff8103b33f>] warn_slowpath_common+0x7f/0xc0
[34173.893417]  [<ffffffff8103b39a>] warn_slowpath_null+0x1a/0x20
[34173.893424]  [<ffffffff8159654c>] mutex_lock_nested+0x39c/0x3b0
[34173.893432]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
[34173.893441]  [<ffffffff8106bded>] ? __blocking_notifier_call_chain+0x7d/0xd0
[34173.893449]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
[34173.893457]  [<ffffffff81074ce1>] ? get_parent_ip+0x11/0x50
[34173.893464]  [<ffffffff81074d99>] ? sub_preempt_count+0x79/0xd0
[34173.893472]  [<ffffffff8144b94d>] cpufreq_governor_dbs+0x3bd/0x560
[34173.893480]  [<ffffffff8144b24a>] od_cpufreq_governor_dbs+0x1a/0x20
[34173.893487]  [<ffffffff81448f13>] __cpufreq_governor+0x53/0xf0
[34173.893494]  [<ffffffff814494a5>] __cpufreq_set_policy+0x155/0x180
[34173.893502]  [<ffffffff8144a483>] cpufreq_update_policy+0xf3/0x130
[34173.893510]  [<ffffffff8144a4c0>] ? cpufreq_update_policy+0x130/0x130
[34173.893519]  [<ffffffff8144a4d1>] handle_update+0x11/0x20
[34173.893526]  [<ffffffff8105f3f7>] process_one_work+0x1f7/0x670
[34173.893533]  [<ffffffff8105f38c>] ? process_one_work+0x18c/0x670
[34173.893541]  [<ffffffff8105fbfe>] worker_thread+0x10e/0x370
[34173.893548]  [<ffffffff8105faf0>] ? rescuer_thread+0x240/0x240
[34173.893556]  [<ffffffff810654fb>] kthread+0xdb/0xe0
[34173.893563]  [<ffffffff81071ba5>] ? finish_task_switch+0x85/0x110
[34173.893572]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
[34173.893579]  [<ffffffff8159a71c>] ret_from_fork+0x7c/0xb0
[34173.893587]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
[34173.893594] ---[ end trace 66f5addf492b41b2 ]---
Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#27 lore raw

Re: More cpufreq breakage

Quoted 3 lines · Borislav Petkov

Btw,

while we're at it, I just discovered this in dmesg. Plain -rc3.

Forgot to say, this happened once during resume.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#28 lore raw

Re: More cpufreq breakage

In reply to Borislav Petkov
Quoted 3 lines · Borislav Petkov

Btw,

while we're at it, I just discovered this in dmesg. Plain -rc3.

I am looking at it since some time, but need a bit of help.

Quoted 6 lines
[34173.893305] ------------[ cut here ]------------
[34173.893321] WARNING: at kernel/mutex.c:199 mutex_lock_nested+0x39c/0x3b0()
[34173.893328] Hardware name: To be filled by O.E.M.
[34173.893333] Modules linked in: nls_iso8859_15 nls_cp437 fuse tun cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative dm_crypt dm_mod ipv6 vfat fat acpi_cpufreq mperf kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 ablk_helper cryptd xts lrw gf128mul microcode radeon amd64_edac_mod edac_core k10temp fam15h_power drm_kms_helper ttm cfbfillrect cfbimgblt r8169 cfbcopyarea
[34173.893395] Pid: 15316, comm: kworker/0:0 Not tainted 3.9.0-rc3 #1
[34173.893402] Call Trace:
Quoted 8 lines
[34173.893409]  [<ffffffff8103b33f>] warn_slowpath_common+0x7f/0xc0
[34173.893417]  [<ffffffff8103b39a>] warn_slowpath_null+0x1a/0x20
[34173.893424]  [<ffffffff8159654c>] mutex_lock_nested+0x39c/0x3b0
[34173.893432]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
[34173.893441]  [<ffffffff8106bded>] ? __blocking_notifier_call_chain+0x7d/0xd0
[34173.893449]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
[34173.893457]  [<ffffffff81074ce1>] ? get_parent_ip+0x11/0x50
[34173.893464]  [<ffffffff81074d99>] ? sub_preempt_count+0x79/0xd0

What does "?" mean before a functions name in trace? I am not able to figure out the exact sequence as i don't see why sub_preempt_count() and get_parent_ip() got called. And then this notifier call.

author maintainer
#29 lore raw

Re: More cpufreq breakage

Quoted 6 lines · Viresh Kumar
Quoted 3 lines · Borislav Petkov

Btw,

while we're at it, I just discovered this in dmesg. Plain -rc3.

I am looking at it since some time, but need a bit of help.

Can you also give us what prints you got before this happened? That would be helpful..

#30 lore raw

Re: More cpufreq breakage

In reply to Viresh Kumar
Quoted 1 line · Viresh Kumar

What does "?" mean before a functions name in trace?

http://stackoverflow.com/questions/13113384/the-meaning-of-in-linux-kernel-panic-call-trace

Here's what's in dmesg before that (basically the machine is hibernating):

Mar 20 10:14:40 pd vmunix: [34172.600018] PM: Syncing filesystems ... done.
Mar 20 10:14:40 pd vmunix: [34172.606180] Freezing user space processes ... (elapsed 0.01 seconds) done.
Mar 20 10:14:40 pd vmunix: [34172.620559] PM: Preallocating image memory... done (allocated 573862 pages)
Mar 20 10:14:40 pd vmunix: [34173.370592] PM: Allocated 2295448 kbytes in 0.75 seconds (3060.59 MB/s)
Mar 20 10:14:40 pd vmunix: [34173.370615] Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
Mar 20 10:14:40 pd vmunix: [34173.388082] serial 00:09: disabled
Mar 20 10:14:40 pd vmunix: [34173.388117] serial 00:09: System wakeup disabled by ACPI
Mar 20 10:14:40 pd vmunix: [34173.695941] PM: freeze of devices complete after 311.852 msecs
Mar 20 10:14:40 pd vmunix: [34173.696673] PM: late freeze of devices complete after 0.722 msecs
Mar 20 10:14:40 pd vmunix: [34173.698914] PM: noirq freeze of devices complete after 2.234 msecs
Mar 20 10:14:40 pd vmunix: [34173.698932] Disabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.703351] smpboot: CPU 1 is now offline
Mar 20 10:14:40 pd vmunix: [34173.708298] smpboot: CPU 2 is now offline
Mar 20 10:14:40 pd vmunix: [34173.712056] smpboot: CPU 3 is now offline
Mar 20 10:14:40 pd vmunix: [34173.717512] smpboot: CPU 4 is now offline
Mar 20 10:14:40 pd vmunix: [34173.722044] smpboot: CPU 5 is now offline
Mar 20 10:14:40 pd vmunix: [34173.728418] smpboot: CPU 6 is now offline
Mar 20 10:14:40 pd vmunix: [34173.731577] smpboot: CPU 7 is now offline
Mar 20 10:14:40 pd vmunix: [34173.732828] PM: Creating hibernation image:
Mar 20 10:14:40 pd vmunix: [34174.371451] PM: Need to copy 596216 pages
Mar 20 10:14:40 pd vmunix: [34173.879106] Enabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.879171] SMP alternatives: lockdep: fixing up alternatives
Mar 20 10:14:40 pd vmunix: [34173.879182] smpboot: Booting Node 0 Processor 1 APIC 0x11
Mar 20 10:14:40 pd vmunix: [34173.890627] LVT offset 0 assigned for vector 0x400
Mar 20 10:14:40 pd vmunix: [34173.893305] ------------[ cut here ]------------
Mar 20 10:14:40 pd vmunix: [34173.893321] WARNING: at kernel/mutex.c:199 mutex_lock_nested+0x39c/0x3b0()
Mar 20 10:14:40 pd vmunix: [34173.893328] Hardware name: To be filled by O.E.M.
Mar 20 10:14:40 pd vmunix: [34173.893333] Modules linked in: nls_iso8859_15 nls_cp437 fuse tun cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative dm_crypt dm_mod ipv6 vfat fat acpi_cpufreq mperf kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 ablk_helper cryptd xts lrw gf128mul microcode radeon amd64_edac_mod edac_core k10temp fam15h_power drm_kms_helper ttm cfbfillrect cfbimgblt r8169 cfbcopyarea
Mar 20 10:14:40 pd vmunix: [34173.893395] Pid: 15316, comm: kworker/0:0 Not tainted 3.9.0-rc3 #1
Mar 20 10:14:40 pd vmunix: [34173.893402] Call Trace:
Mar 20 10:14:40 pd vmunix: [34173.893409]  [<ffffffff8103b33f>] warn_slowpath_common+0x7f/0xc0
Mar 20 10:14:40 pd vmunix: [34173.893417]  [<ffffffff8103b39a>] warn_slowpath_null+0x1a/0x20
Mar 20 10:14:40 pd vmunix: [34173.893424]  [<ffffffff8159654c>] mutex_lock_nested+0x39c/0x3b0
Mar 20 10:14:40 pd vmunix: [34173.893432]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893441]  [<ffffffff8106bded>] ? __blocking_notifier_call_chain+0x7d/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893449]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893457]  [<ffffffff81074ce1>] ? get_parent_ip+0x11/0x50
Mar 20 10:14:40 pd vmunix: [34173.893464]  [<ffffffff81074d99>] ? sub_preempt_count+0x79/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893472]  [<ffffffff8144b94d>] cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893480]  [<ffffffff8144b24a>] od_cpufreq_governor_dbs+0x1a/0x20
Mar 20 10:14:40 pd vmunix: [34173.893487]  [<ffffffff81448f13>] __cpufreq_governor+0x53/0xf0
Mar 20 10:14:40 pd vmunix: [34173.893494]  [<ffffffff814494a5>] __cpufreq_set_policy+0x155/0x180
Mar 20 10:14:40 pd vmunix: [34173.893502]  [<ffffffff8144a483>] cpufreq_update_policy+0xf3/0x130
Mar 20 10:14:40 pd vmunix: [34173.893510]  [<ffffffff8144a4c0>] ? cpufreq_update_policy+0x130/0x130
Mar 20 10:14:40 pd vmunix: [34173.893519]  [<ffffffff8144a4d1>] handle_update+0x11/0x20
Mar 20 10:14:40 pd vmunix: [34173.893526]  [<ffffffff8105f3f7>] process_one_work+0x1f7/0x670
Mar 20 10:14:40 pd vmunix: [34173.893533]  [<ffffffff8105f38c>] ? process_one_work+0x18c/0x670
Mar 20 10:14:40 pd vmunix: [34173.893541]  [<ffffffff8105fbfe>] worker_thread+0x10e/0x370
Mar 20 10:14:40 pd vmunix: [34173.893548]  [<ffffffff8105faf0>] ? rescuer_thread+0x240/0x240
Mar 20 10:14:40 pd vmunix: [34173.893556]  [<ffffffff810654fb>] kthread+0xdb/0xe0
Mar 20 10:14:40 pd vmunix: [34173.893563]  [<ffffffff81071ba5>] ? finish_task_switch+0x85/0x110
Mar 20 10:14:40 pd vmunix: [34173.893572]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
Mar 20 10:14:40 pd vmunix: [34173.893579]  [<ffffffff8159a71c>] ret_from_fork+0x7c/0xb0
Mar 20 10:14:40 pd vmunix: [34173.893587]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
Mar 20 10:14:40 pd vmunix: [34173.893594] ---[ end trace 66f5addf492b41b2 ]---

HTH.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#31 lore raw

Re: More cpufreq breakage

Quoted 55 lines · Borislav Petkov
Quoted 1 line · Viresh Kumar

What does "?" mean before a functions name in trace?

http://stackoverflow.com/questions/13113384/the-meaning-of-in-linux-kernel-panic-call-trace

Here's what's in dmesg before that (basically the machine is hibernating):

Mar 20 10:14:40 pd vmunix: [34172.600018] PM: Syncing filesystems ... done.
Mar 20 10:14:40 pd vmunix: [34172.606180] Freezing user space processes ... (elapsed 0.01 seconds) done.
Mar 20 10:14:40 pd vmunix: [34172.620559] PM: Preallocating image memory... done (allocated 573862 pages)
Mar 20 10:14:40 pd vmunix: [34173.370592] PM: Allocated 2295448 kbytes in 0.75 seconds (3060.59 MB/s)
Mar 20 10:14:40 pd vmunix: [34173.370615] Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
Mar 20 10:14:40 pd vmunix: [34173.388082] serial 00:09: disabled
Mar 20 10:14:40 pd vmunix: [34173.388117] serial 00:09: System wakeup disabled by ACPI
Mar 20 10:14:40 pd vmunix: [34173.695941] PM: freeze of devices complete after 311.852 msecs
Mar 20 10:14:40 pd vmunix: [34173.696673] PM: late freeze of devices complete after 0.722 msecs
Mar 20 10:14:40 pd vmunix: [34173.698914] PM: noirq freeze of devices complete after 2.234 msecs
Mar 20 10:14:40 pd vmunix: [34173.698932] Disabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.703351] smpboot: CPU 1 is now offline
Mar 20 10:14:40 pd vmunix: [34173.708298] smpboot: CPU 2 is now offline
Mar 20 10:14:40 pd vmunix: [34173.712056] smpboot: CPU 3 is now offline
Mar 20 10:14:40 pd vmunix: [34173.717512] smpboot: CPU 4 is now offline
Mar 20 10:14:40 pd vmunix: [34173.722044] smpboot: CPU 5 is now offline
Mar 20 10:14:40 pd vmunix: [34173.728418] smpboot: CPU 6 is now offline
Mar 20 10:14:40 pd vmunix: [34173.731577] smpboot: CPU 7 is now offline
Mar 20 10:14:40 pd vmunix: [34173.732828] PM: Creating hibernation image:
Mar 20 10:14:40 pd vmunix: [34174.371451] PM: Need to copy 596216 pages
Mar 20 10:14:40 pd vmunix: [34173.879106] Enabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.879171] SMP alternatives: lockdep: fixing up alternatives
Mar 20 10:14:40 pd vmunix: [34173.879182] smpboot: Booting Node 0 Processor 1 APIC 0x11
Mar 20 10:14:40 pd vmunix: [34173.890627] LVT offset 0 assigned for vector 0x400
Mar 20 10:14:40 pd vmunix: [34173.893305] ------------[ cut here ]------------
Mar 20 10:14:40 pd vmunix: [34173.893321] WARNING: at kernel/mutex.c:199 mutex_lock_nested+0x39c/0x3b0()
Mar 20 10:14:40 pd vmunix: [34173.893328] Hardware name: To be filled by O.E.M.
Mar 20 10:14:40 pd vmunix: [34173.893333] Modules linked in: nls_iso8859_15 nls_cp437 fuse tun cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative dm_crypt dm_mod ipv6 vfat fat acpi_cpufreq mperf kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 ablk_helper cryptd xts lrw gf128mul microcode radeon amd64_edac_mod edac_core k10temp fam15h_power drm_kms_helper ttm cfbfillrect cfbimgblt r8169 cfbcopyarea
Mar 20 10:14:40 pd vmunix: [34173.893395] Pid: 15316, comm: kworker/0:0 Not tainted 3.9.0-rc3 #1
Mar 20 10:14:40 pd vmunix: [34173.893402] Call Trace:
Mar 20 10:14:40 pd vmunix: [34173.893409]  [<ffffffff8103b33f>] warn_slowpath_common+0x7f/0xc0
Mar 20 10:14:40 pd vmunix: [34173.893417]  [<ffffffff8103b39a>] warn_slowpath_null+0x1a/0x20
Mar 20 10:14:40 pd vmunix: [34173.893424]  [<ffffffff8159654c>] mutex_lock_nested+0x39c/0x3b0
Mar 20 10:14:40 pd vmunix: [34173.893432]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893441]  [<ffffffff8106bded>] ? __blocking_notifier_call_chain+0x7d/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893449]  [<ffffffff8144b94d>] ? cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893457]  [<ffffffff81074ce1>] ? get_parent_ip+0x11/0x50
Mar 20 10:14:40 pd vmunix: [34173.893464]  [<ffffffff81074d99>] ? sub_preempt_count+0x79/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893472]  [<ffffffff8144b94d>] cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893480]  [<ffffffff8144b24a>] od_cpufreq_governor_dbs+0x1a/0x20
Mar 20 10:14:40 pd vmunix: [34173.893487]  [<ffffffff81448f13>] __cpufreq_governor+0x53/0xf0
Mar 20 10:14:40 pd vmunix: [34173.893494]  [<ffffffff814494a5>] __cpufreq_set_policy+0x155/0x180
Mar 20 10:14:40 pd vmunix: [34173.893502]  [<ffffffff8144a483>] cpufreq_update_policy+0xf3/0x130
Mar 20 10:14:40 pd vmunix: [34173.893510]  [<ffffffff8144a4c0>] ? cpufreq_update_policy+0x130/0x130
Mar 20 10:14:40 pd vmunix: [34173.893519]  [<ffffffff8144a4d1>] handle_update+0x11/0x20
Mar 20 10:14:40 pd vmunix: [34173.893526]  [<ffffffff8105f3f7>] process_one_work+0x1f7/0x670
Mar 20 10:14:40 pd vmunix: [34173.893533]  [<ffffffff8105f38c>] ? process_one_work+0x18c/0x670
Quoted 8 lines
Mar 20 10:14:40 pd vmunix: [34173.893541]  [<ffffffff8105fbfe>] worker_thread+0x10e/0x370
Mar 20 10:14:40 pd vmunix: [34173.893548]  [<ffffffff8105faf0>] ? rescuer_thread+0x240/0x240
Mar 20 10:14:40 pd vmunix: [34173.893556]  [<ffffffff810654fb>] kthread+0xdb/0xe0
Mar 20 10:14:40 pd vmunix: [34173.893563]  [<ffffffff81071ba5>] ? finish_task_switch+0x85/0x110
Mar 20 10:14:40 pd vmunix: [34173.893572]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
Mar 20 10:14:40 pd vmunix: [34173.893579]  [<ffffffff8159a71c>] ret_from_fork+0x7c/0xb0
Mar 20 10:14:40 pd vmunix: [34173.893587]  [<ffffffff81065420>] ? __init_kthread_worker+0x70/0x70
Mar 20 10:14:40 pd vmunix: [34173.893594] ---[ end trace 66f5addf492b41b2 ]---

This looks like the CPU offline path to me (i.e. the CPU hotplug notifier does something fishy).

Thanks,
Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
author maintainer
#32 lore raw

Re: More cpufreq breakage

Quoted 18 lines · Rafael J. Wysocki
Quoted 8 lines
Mar 20 10:14:40 pd vmunix: [34173.698932] Disabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.703351] smpboot: CPU 1 is now offline
Mar 20 10:14:40 pd vmunix: [34173.708298] smpboot: CPU 2 is now offline
Mar 20 10:14:40 pd vmunix: [34173.712056] smpboot: CPU 3 is now offline
Mar 20 10:14:40 pd vmunix: [34173.717512] smpboot: CPU 4 is now offline
Mar 20 10:14:40 pd vmunix: [34173.722044] smpboot: CPU 5 is now offline
Mar 20 10:14:40 pd vmunix: [34173.728418] smpboot: CPU 6 is now offline
Mar 20 10:14:40 pd vmunix: [34173.731577] smpboot: CPU 7 is now offline
Quoted 8 lines
Mar 20 10:14:40 pd vmunix: [34173.732828] PM: Creating hibernation image:
Mar 20 10:14:40 pd vmunix: [34174.371451] PM: Need to copy 596216 pages
Mar 20 10:14:40 pd vmunix: [34173.879106] Enabling non-boot CPUs ...
Mar 20 10:14:40 pd vmunix: [34173.879171] SMP alternatives: lockdep: fixing up alternatives
Mar 20 10:14:40 pd vmunix: [34173.879182] smpboot: Booting Node 0 Processor 1 APIC 0x11
Mar 20 10:14:40 pd vmunix: [34173.890627] LVT offset 0 assigned for vector 0x400
Mar 20 10:14:40 pd vmunix: [34173.893305] ------------[ cut here ]------------
Mar 20 10:14:40 pd vmunix: [34173.893321] WARNING: at kernel/mutex.c:199 mutex_lock_nested+0x39c/0x3b0()
Quoted 2 lines

This looks like the CPU offline path to me (i.e. the CPU hotplug notifier does something fishy).

I think otherwise, Its the cpu online path but this didn't happened for first boot (probably).

Enabling non-boot CPUs ...

--
viresh

author maintainer
#33 lore raw

Re: More cpufreq breakage

Quoted 2 lines · Viresh Kumar

I think otherwise, Its the cpu online path but this didn't happened for first boot (probably).

I tried on my 4 cpu laptop and my bad, couldn't reproduce the issue reported by both Borislav and Duncan :(

Hibernation logs (Borislav's bug):
https://pastebin.linaro.org/2019/

cpufreq-info after hibernation (same happens with suspend) (Duncan's bug):
https://pastebin.linaro.org/2020/

The main difference between our systems is number of cpu groups that share clock line. On setup of both Duncan and Borislav, they had total of 8 cpus and four groups 0-1, 2-3, 4-5, 6-7. And thus have four policy structures. And i have only one group 0-1-2-3 and thus only one policy struct.

For Duncan: The first policy structure (that has boot cpu) never gets corrupted and probably that's why i am not able to reproduce it again.

@Borislav: BTW, can you try reproducing your issue again? If that is reproducible?

I am asking for this because your logs looked confusing to me.

Mar 20 10:14:40 pd vmunix: [34173.893409]  [<ffffffff8103b33f>]
warn_slowpath_common+0x7f/0xc0
Mar 20 10:14:40 pd vmunix: [34173.893417]  [<ffffffff8103b39a>]
warn_slowpath_null+0x1a/0x20
Mar 20 10:14:40 pd vmunix: [34173.893424]  [<ffffffff8159654c>]
mutex_lock_nested+0x39c/0x3b0
Mar 20 10:14:40 pd vmunix: [34173.893432]  [<ffffffff8144b94d>] ?
cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893441]  [<ffffffff8106bded>] ?
__blocking_notifier_call_chain+0x7d/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893449]  [<ffffffff8144b94d>] ?
cpufreq_governor_dbs+0x3bd/0x560
Mar 20 10:14:40 pd vmunix: [34173.893457]  [<ffffffff81074ce1>] ?
get_parent_ip+0x11/0x50
Mar 20 10:14:40 pd vmunix: [34173.893464]  [<ffffffff81074d99>] ?
sub_preempt_count+0x79/0xd0
Mar 20 10:14:40 pd vmunix: [34173.893472]  [<ffffffff8144b94d>]

cpufreq_governor_dbs+0x3bd/0x560

I don't see (logically) how sub_preempt_count() can be called from cpufreq_governor_dbs()? As it is mostly called from kernel/sched/ part only.

So maybe two logs are mixed here and crash got due to sub_preempt_count()->get_parent_ip() and not cpufreq. :)

If you still get it, try disabling cpufreq completely and see if it is gone or not.

Thanks in Advance.

--
viresh

#34 lore raw

Re: More cpufreq breakage

Quoted 4 lines · Viresh Kumar
Quoted 2 lines · Viresh Kumar

I think otherwise, Its the cpu online path but this didn't happened for first boot (probably).

Quoted 8 lines

I tried on my 4 cpu laptop and my bad, couldn't reproduce the issue reported by both Borislav and Duncan :(

Hibernation logs (Borislav's bug):
https://pastebin.linaro.org/2019/

cpufreq-info after hibernation (same happens with suspend) (Duncan's bug):
https://pastebin.linaro.org/2020/

Those pastebin things want a login. Use a free one.

Quoted 5 lines

The main difference between our systems is number of cpu groups that share clock line. On setup of both Duncan and Borislav, they had total of 8 cpus and four groups 0-1, 2-3, 4-5, 6-7. And thus have four policy structures. And i have only one group 0-1-2-3 and thus only one policy struct.

So this should give you a clue - you need to repro it on a similar machine and your laptop is obviously not similar.

Quoted 2 lines

@Borislav: BTW, can you try reproducing your issue again? If that is reproducible?

I've seen it only once so far and I've suspended the machine a bunch of times already. So I don't think it is that easy to reproduce.

Quoted 3 lines

I don't see (logically) how sub_preempt_count() can be called from cpufreq_governor_dbs()? As it is mostly called from kernel/sched/ part only.

As Rafael said, there's a notifier running which can, AFAICT, disable preemption on another CPU in parallel, for example.

Quoted 2 lines

If you still get it, try disabling cpufreq completely and see if it is gone or not.

Unfortunately this is my desktop machine and I don't want to test stuff on it because I need it to work. And I've already downgraded to 3.8.3 because of the other cpufreq breakage which kept a subset of the cores at max freq because acpi-cpufreq wasn't loading on them.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#35 lore raw

Re: More cpufreq breakage

Quoted 2 lines · Borislav Petkov

On Sat, Mar 23, 2013 at 08:46:00PM +0530, Viresh Kumar wrote:
Those pastebin things want a login. Use a free one.

Attached now.

Quoted 2 lines

So this should give you a clue - you need to repro it on a similar machine and your laptop is obviously not similar.

I know that. I would be trying to on monday on my office desktop.

Quoted 4 lines

Unfortunately this is my desktop machine and I don't want to test stuff on it because I need it to work. And I've already downgraded to 3.8.3 because of the other cpufreq breakage which kept a subset of the cores at max freq because acpi-cpufreq wasn't loading on them.

And that is already caused recently?

--
viresh

viresh@blr-linut-001:$ cpufreq-info
cpufrequtils 007: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 10.0 us.
  hardware limits: 800 MHz - 2.80 GHz
  available frequency steps: 2.80 GHz, 2.80 GHz, 2.60 GHz, 2.40 GHz, 2.20 GHz, 2.00 GHz, 1.80 GHz, 1.60 GHz, 1.40 GHz, 1.20 GHz, 1000 MHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 2.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 2.80 GHz:8.64%, 2.80 GHz:0.16%, 2.60 GHz:0.13%, 2.40 GHz:0.10%, 2.20 GHz:0.08%, 2.00 GHz:0.12%, 1.80 GHz:0.08%, 1.60 GHz:0.24%, 1.40 GHz:0.12%, 1.20 GHz:0.17%, 1000 MHz:0.28%, 800 MHz:89.87%  (1943)
analyzing CPU 1:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 10.0 us.
  hardware limits: 800 MHz - 2.80 GHz
  available frequency steps: 2.80 GHz, 2.80 GHz, 2.60 GHz, 2.40 GHz, 2.20 GHz, 2.00 GHz, 1.80 GHz, 1.60 GHz, 1.40 GHz, 1.20 GHz, 1000 MHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 2.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 2.80 GHz:8.64%, 2.80 GHz:0.16%, 2.60 GHz:0.13%, 2.40 GHz:0.10%, 2.20 GHz:0.08%, 2.00 GHz:0.12%, 1.80 GHz:0.08%, 1.60 GHz:0.24%, 1.40 GHz:0.12%, 1.20 GHz:0.17%, 1000 MHz:0.28%, 800 MHz:89.87%  (1943)
analyzing CPU 2:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 10.0 us.
  hardware limits: 800 MHz - 2.80 GHz
  available frequency steps: 2.80 GHz, 2.80 GHz, 2.60 GHz, 2.40 GHz, 2.20 GHz, 2.00 GHz, 1.80 GHz, 1.60 GHz, 1.40 GHz, 1.20 GHz, 1000 MHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 2.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 2.80 GHz:8.64%, 2.80 GHz:0.16%, 2.60 GHz:0.13%, 2.40 GHz:0.10%, 2.20 GHz:0.08%, 2.00 GHz:0.12%, 1.80 GHz:0.08%, 1.60 GHz:0.24%, 1.40 GHz:0.12%, 1.20 GHz:0.17%, 1000 MHz:0.28%, 800 MHz:89.87%  (1943)
analyzing CPU 3:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 10.0 us.
  hardware limits: 800 MHz - 2.80 GHz
  available frequency steps: 2.80 GHz, 2.80 GHz, 2.60 GHz, 2.40 GHz, 2.20 GHz, 2.00 GHz, 1.80 GHz, 1.60 GHz, 1.40 GHz, 1.20 GHz, 1000 MHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 2.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 2.80 GHz:8.64%, 2.80 GHz:0.16%, 2.60 GHz:0.13%, 2.40 GHz:0.10%, 2.20 GHz:0.08%, 2.00 GHz:0.12%, 1.80 GHz:0.08%, 1.60 GHz:0.24%, 1.40 GHz:0.12%, 1.20 GHz:0.17%, 1000 MHz:0.28%, 800 MHz:89.87%  (1943)
#36 lore raw

Re: More cpufreq breakage

Quoted 1 line · Viresh Kumar

And that is already caused recently?

3.8.3 is fine and there are no cpufreq patches in v3.8..v3.8.3. Which would mean, plain v3.8 is also fine. Which means, the breakage must've come in during the merge window.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#37 lore raw

Re: More cpufreq breakage

[Forgot to add all in previous mail, reply-all now]

Quoted 2 lines · Borislav Petkov
Quoted 1 line · Viresh Kumar

And that is already caused recently?

Ahh!! s/already/also

Quoted 3 lines

3.8.3 is fine and there are no cpufreq patches in v3.8..v3.8.3. Which would mean, plain v3.8 is also fine. Which means, the breakage must've come in during the merge window.

I didn't get a complete picture of your problem, can you elaborate a bit with all your observations?

It might be related to the problem reported by Duncan and might be solved by the patch i sent yesterday.

#38 lore raw

Re: More cpufreq breakage

Quoted 2 lines · Viresh Kumar

I didn't get a complete picture of your problem, can you elaborate a bit with all your observations?

It is basically the same observation. Some of the even cores are not handled by any cpufreq driver anymore, i.e. in the cpufreq-info output. And thus they remain in P0 which is max freq. And this happens because I'm suspending my workstation at night which takes the cores offline, so it is the same trigger as Duncan's.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#39 lore raw
In reply to Viresh Kumar
Quoted 2 lines · Viresh Kumar

We are talking here about a bug reported by Duncan here. His cpu/cpu*/cpufreq directory are getting corrupted with 3.9-rc3 and was working well with 3.8

Hi Duncan,

My weekend is already spoiled (due to my bugs :) ) and i don't want to spoil yours. But this is a important patch to test, can you please give it a try as soon
as you can? And confirm if this resolves your issues or not? (patch is attached too)

------------x------------------x------------------

From: Viresh Kumar <[email protected]>
Date: Sat, 23 Mar 2013 23:42:44 +0530
Subject: [PATCH] cpufreq: acpi-cpufreq: Set policy->cpus correctly from .init()

With the addition of following patch:

fcf8058 cpufreq: Simplify cpufreq_add_dev()

cpufreq driver's .init() routine must initialize policy->cpus with mask of all possible cpus (Online + Offline) that share the clock. Then the core would copy this mask onto policy->related_cpus and will reset policy->cpus to carry only online cpus.

acpi-cpufreq driver wasn't updated with this assumption and so sometimes when we try to hot[un]plug cpus at run time, sysfs directories gets corrupted.

This patch fixes acpi-cpufreq driver against this corruption.

---
 drivers/cpufreq/acpi-cpufreq.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
drivers/cpufreq/acpi-cpufreq.c +0 -0
index 937bc28..ceb5925 100644
@@ -722,15 +722,7 @@ static int acpi_cpufreq_cpu_init(struct

cpufreq_policy *policy)
perf = data->acpi_data;
policy->shared_type = perf->shared_type;

- /*
- * Will let policy->cpus know about dependency only when software
- * coordination is required.
- */
- if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL ||
- policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) {
- cpumask_copy(policy->cpus, perf->shared_cpu_map);
- }
- cpumask_copy(policy->related_cpus, perf->shared_cpu_map);
+ cpumask_copy(policy->cpus, perf->shared_cpu_map);

#ifdef CONFIG_SMP
dmi_check_system(sw_any_bug_dmi_table);
@@ -740,9 +732,7 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
}

if (check_amd_hwpstate_cpu(cpu) && !acpi_pstate_strict) {
- cpumask_clear(policy->cpus);
- cpumask_set_cpu(cpu, policy->cpus);
- cpumask_copy(policy->related_cpus, cpu_sibling_mask(cpu));
+ cpumask_copy(policy->cpus, cpu_sibling_mask(cpu));
policy->shared_type = CPUFREQ_SHARED_TYPE_HW; pr_info_once(PFX "overriding BIOS provided _PSD data\n");
}

--
viresh

From 4fb03dd59965d8fa011df49358632f95aef1fa74 Mon Sep 17 00:00:00 2001
Message-Id: <4fb03dd59965d8fa011df49358632f95aef1fa74.1364062607.git.viresh.kumar@linaro.org>
From: Viresh Kumar <[email protected]>
Date: Sat, 23 Mar 2013 23:42:44 +0530
Subject: [PATCH] cpufreq: acpi-cpufreq: Set policy->cpus correctly from
.init()

With the addition of following patch:

fcf8058 cpufreq: Simplify cpufreq_add_dev()

cpufreq driver's .init() routine must initialize policy->cpus with mask of all possible cpus (Online + Offline) that share the clock. Then the core would copy this mask onto policy->related_cpus and will reset policy->cpus to carry only online cpus.

acpi-cpufreq driver wasn't updated with this assumption and so sometimes when we try to hot[un]plug cpus at run time, sysfs directories gets corrupted.

This patch fixes acpi-cpufreq driver against this corruption.

---
 drivers/cpufreq/acpi-cpufreq.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
drivers/cpufreq/acpi-cpufreq.c +2 -12
index 937bc28..ceb5925 100644
@@ -722,15 +722,7 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	perf = data->acpi_data;
 	policy->shared_type = perf->shared_type;
 
-	/*
-	 * Will let policy->cpus know about dependency only when software
-	 * coordination is required.
-	 */
-	if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL ||
-	    policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) {
-		cpumask_copy(policy->cpus, perf->shared_cpu_map);
-	}
-	cpumask_copy(policy->related_cpus, perf->shared_cpu_map);
+	cpumask_copy(policy->cpus, perf->shared_cpu_map);
 
 #ifdef CONFIG_SMP
 	dmi_check_system(sw_any_bug_dmi_table);
@@ -740,9 +732,7 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	}
 
 	if (check_amd_hwpstate_cpu(cpu) && !acpi_pstate_strict) {
-		cpumask_clear(policy->cpus);
-		cpumask_set_cpu(cpu, policy->cpus);
-		cpumask_copy(policy->related_cpus, cpu_sibling_mask(cpu));
+		cpumask_copy(policy->cpus, cpu_sibling_mask(cpu));
 		policy->shared_type = CPUFREQ_SHARED_TYPE_HW;
 		pr_info_once(PFX "overriding BIOS provided _PSD data\n");
 	}
Signature · 1 line
1.7.12.rc2.18.g61b472e