Hotplug cpu: fix p4-clockmod
Quoted 2 lines
This currently oopses when a processor goes offline since it can't schedule itself onto the correct cpu.
What's the call trace? Because I don't see why it should happen: if the CPU is removed, cpufreq_remove_dev() gets called, which cleans up the governor and the device driver -- and I can't see a call to setdc in this code path.
Quoted 1 line
So we now disable preemption whilst reading or setting the DC.
Or should we disable CPU hotplug [IIRC there's some callback to temporarily do that] in the cpufreq core before calling ->target() or ->get()
Quoted 2 lines
There is a questionable hunk which returns the
stock_freqif the processor being read from is offline.
and it would avoid this very very questionable hunk...
Thanks,
Dominik
Quoted 6 lines · Dominik Brodowski
Quoted 2 lines
This currently oopses when a processor goes offline since it can't schedule itself onto the correct cpu.
What's the call trace? Because I don't see why it should happen: if the CPU is removed,
cpufreq_remove_dev()gets called, which cleans up the governor and the device driver -- and I can't see a call to setdc in this code path.
kernel BUG at arch/i386/kernel/cpu/cpufreq/p4-clockmod.c:148! invalid operand: 0000 [#1]
PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: CPU: 0 EIP: 0060:[<c01147a3>] Not tainted VLI EFLAGS: 00010297 (2.6.8.1-mm2)
EIP is at cpufreq_p4_target+0x1c3/0x210
eax: eb3b4a40 ebx: eb365e98 ecx: eb3b4a40 edx: 00000002
esi: eb364000 edi: 00000001 ebp: eb365ebc esp: eb365e94
ds: 007b es: 007b ss: 0068
Process cpuspeed (pid: 2491, threadinfo=eb364000 task=eb3b4a40)
Stack: 00000003 00000003 00000007 00000001 0030d400 002ab980 eb365f02 c06d8a40
00000000 c0832ca0 eb365edc c0557199 c0832ca0 002ab980 00000000 002ab980
00000000 c073f240 eb365f00 c0557ec2 c0832ca0 002ab980 00000000 ffffffea
Call Trace:
[<c0108915>] show_stack+0x75/0x90
[<c0108a75>] show_registers+0x125/0x190
[<c0108c79>] die+0x109/0x1f0
[<c0109153>] do_invalid_op+0xe3/0xf0
[<c01084dd>] error_code+0x2d/0x40
[<c0557199>] __cpufreq_driver_target+0x69/0x70
[<c0557ec2>] cpufreq_set+0x72/0xa0
[<c05581f7>] store_speed+0x37/0x40
[<c05567cb>] store+0x4b/0x60
[<c01b2178>] flush_write_buffer+0x28/0x30
[<c01b21be>] sysfs_write_file+0x3e/0x50
[<c016dbdc>] vfs_write+0xac/0x100
[<c016dcdc>] sys_write+0x3c/0x70
[<c0107359>] sysenter_past_esp+0x52/0x79
Code: ff ff 59 58 8d 47 01 50 6a 02 53 e8 18 26 28 00 83 c4 0c 89 c1 83 f8 02 7e 05 b9 02 00 00 00 89 cf 83 f9 01 7e b2 e9 22 ff ff ff <0f> 0b 94 00 70 9d 64 c0 eb bc 8d 76 00 8d 75 e4 6a 00 89 7d e4Quoted 6 lines
Quoted 1 line
So we now disable preemption whilst reading or setting the DC.
Or should we disable CPU hotplug [IIRC there's some callback to temporarily do that] in the cpufreq core before calling ->target() or ->get()
Quoted 2 lines
There is a questionable hunk which returns the
stock_freqif the processor being read from is offline.
The cpufreq_sysdev_driver stuff only covers device removal and won't cover the processor being removed. In order to take care of that you need to either disable preemption or hold the cpucontrol semaphore, or you could register a cpu online/offline notifier and do the necessary work there.
Thanks,
Zwane
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Dominik Brodowski
Quoted 2 lines
This currently oopses when a processor goes offline since it can't schedule itself onto the correct cpu.
What's the call trace? Because I don't see why it should happen: if the CPU is removed,
cpufreq_remove_dev()gets called, which cleans up the governor and the device driver -- and I can't see a call to setdc in this code path.kernel BUG at
arch/i386/kernel/cpu/cpufreq/p4-clockmod.c:148!
Thanks.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
So we now disable preemption whilst reading or setting the DC.
Or should we disable CPU hotplug [IIRC there's some callback to temporarily do that] in the cpufreq core before calling ->target() or ->get()
Quoted 2 lines
There is a questionable hunk which returns the
stock_freqif the processor being read from is offline.The
cpufreq_sysdev_driverstuff only covers device removal and won't cover the processor being removed.
What? If the processor X is removed, the kobject / device of /sys/devices/system/cpu/cpuX/ will not be unregistered? If that is the case, then the current cpufreq core indeed is severely broken....
Quoted 2 lines
In order to take care of that you need to either
disable preemption
... however, I dislike disabling preemption -- it can cause long latencies, after all -- and prefer disabling CPU hotplug around calls to ->target() and ->get()...
Quoted 2 lines
or hold the cpucontrol semaphore, or you could register a cpu online/offline notifier and do the necessary work there.
... and that cpu online/offline notifier must then do what I thought the sysfs device ->add()/->remove() calls were doing.
Thanks,
Dominik
Quoted 6 lines · Dominik Brodowski
Quoted 2 lines
The
cpufreq_sysdev_driverstuff only covers device removal and won't cover the processor being removed.What? If the processor X is removed, the kobject / device of
/sys/devices/system/cpu/cpuX/will not be unregistered? If that is the case, then the current cpufreq core indeed is severely broken....
I think i may be misunderstood, i mean it won't handle a processor being removed from the online map.
Quoted 10 lines
Quoted 2 lines
In order to take care of that you need to either
disable preemption... however, I dislike disabling preemption -- it can cause long latencies, after all -- and prefer disabling CPU hotplug around calls to ->target() and ->get()...
Quoted 2 lines
or hold the cpucontrol semaphore, or you could register a cpu online/offline notifier and do the necessary work there.
... and that cpu online/offline notifier must then do what I thought the sysfs device ->add()/->remove() calls were doing.
It should just be a matter of holding the semaphore during ->get() and ->target, i go for the preempt_disable a lot of the time to avoid too many contenders for the semaphore and since the preemption critical sections are fairly short. Regarding other drivers, yes they are mostly all unsafe, the ACPI one definitely needs looking at too.
Thanks,
Zwane
Quoted 11 lines · Zwane Mwaikambo
Quoted 6 lines · Dominik Brodowski
Quoted 2 lines
The
cpufreq_sysdev_driverstuff only covers device removal and won't cover the processor being removed.What? If the processor X is removed, the kobject / device of
/sys/devices/system/cpu/cpuX/will not be unregistered? If that is the case, then the current cpufreq core indeed is severely broken....I think i may be misunderstood, i mean it won't handle a processor being removed from the online map.
Why? If the CPU is removed [and thus removed from the online map] the device disappears from the kernel's perspective -- so there's no room for the /sys/devices/system/cpu/cpuX/ kobject.
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
In order to take care of that you need to either
disable preemption... however, I dislike disabling preemption -- it can cause long latencies, after all -- and prefer disabling CPU hotplug around calls to ->target() and ->get()...
Quoted 2 lines
or hold the cpucontrol semaphore, or you could register a cpu online/offline notifier and do the necessary work there.
Quoted 7 lines
Quoted 2 lines
... and that cpu online/offline notifier must then do what I thought the sysfs device ->add()/->remove() calls were doing.
It should just be a matter of holding the semaphore during ->get() and ->target, i go for the
preempt_disablea lot of the time to avoid too many contenders for the semaphore and since the preemption critical sections are fairly short.
While it is short for p4-clockmod, it isn't sufficiently short for powernow-k8.
Thanks,
Dominik
Quoted 3 lines · Dominik Brodowski
Why? If the CPU is removed [and thus removed from the online map] the device disappears from the kernel's perspective -- so there's no room for the
/sys/devices/system/cpu/cpuX/kobject.
It hasn't disappeared as a device, only that the processor will not participate in a number of kernel processes. The kobject has to also stay around so that we can online it again.
Quoted 2 lines
While it is short for p4-clockmod, it isn't sufficiently short for powernow-k8.
Yeah we'll have to go case by case.
Thanks,
Zwane
Quoted 9 lines · Zwane Mwaikambo
Quoted 3 lines · Dominik Brodowski
Why? If the CPU is removed [and thus removed from the online map] the device disappears from the kernel's perspective -- so there's no room for the
/sys/devices/system/cpu/cpuX/kobject.It hasn't disappeared as a device, only that the processor will not participate in a number of kernel processes. The kobject has to also stay around so that we can online it again.
OK.
Quoted 4 lines
Quoted 2 lines
While it is short for p4-clockmod, it isn't sufficiently short for powernow-k8.
Yeah we'll have to go case by case.
But I _really_ _don't_ _like_ _it_. Having to disable preemption for the corner case of CPU hotplug. How often do CPU hotplug events happen? Very rarely compared to _other_ valid preempting code, e.g. audio IRQs... And I don't see _any_ reason why CPU hotplug couldn't wait until a cpufreq request for a CPU is finished. However, the current CPU hotplug notifiers don't allow for this: they only notify the notifiers if the CPU is down, not before to avoid migrating to/switching frequency of the dying CPU.
I'm thinking of one of the following two implementations:
A)
DOWN:
1.) notifier informs cpufreq that CPU is dying
2.) cpufreq grabs policy lock
3.) sets a flag or something so that future cpufreq-related calls for this
CPU will fail
4.) cpufreq release policy lock
5.) cpu is removed from online_map ....
UP:
1.) cpu is online
2.) notifier informs cpufreq that CPU is ONLINE
3.) cpufreq grabs policy lock
4.) clears the flag mentioned above
Problem: ->exit() and ->init() aren't called for the CPU going off- or online which means that changes in their cpufreq capabilities won't be noticed.
B)
DOWN:
1.) notifier informs cpufreq that CPU is dying
2.) cpufreq_remove_dev() is executed, finishes only when _all_ current
cpufreq-related work on this CPU is done
3.) cpu is removed from online_map....
UP:
1.) cpu is online
2.) notifier informs cpufreq that CPU is ONLINE
3.) cpufreq_add_dev() is executed
Dominik
Quoted 7 lines · Dominik Brodowski
But I _really_ _don't_ _like_ _it_. Having to disable preemption for the corner case of CPU hotplug. How often do CPU hotplug events happen? Very rarely compared to _other_ valid preempting code, e.g. audio IRQs... And I don't see _any_ reason why CPU hotplug couldn't wait until a cpufreq request for a CPU is finished. However, the current CPU hotplug notifiers don't allow for this: they only notify the notifiers if the CPU is down, not before to avoid migrating to/switching frequency of the dying CPU.
Well interrupts would still preempt (side note, do any of these hardware programming have timing requirements? If so preempt might be a problem anyway) those scheduling preemption disabled sections.
Quoted 17 lines
A)
DOWN:
1.) notifier informs cpufreq that CPU is dying
2.) cpufreq grabs policy lock
3.) sets a flag or something so that future cpufreq-related calls for this
CPU will fail
4.) cpufreq release policy lock
5.) cpu is removed fromonline_map....UP:
1.) cpu is online
2.) notifier informs cpufreq that CPU is ONLINE
3.) cpufreq grabs policy lock
4.) clears the flag mentioned above
Quoted 3 lines
Problem: ->exit() and ->init() aren't called for the CPU going off- or online which means that changes in their cpufreq capabilities won't be noticed.
Well the major differences between A and B appear to be discovery of new capabilities on online processors. I wouldn't go that far as asynchronous multiprocessing systems aren't supported in general, but at least it allows for releasing memory for unused processors, which can't be a bad thing.
Thanks,
Zwane
Quoted 10 lines · Zwane Mwaikambo
Quoted 7 lines · Dominik Brodowski
But I _really_ _don't_ _like_ _it_. Having to disable preemption for the corner case of CPU hotplug. How often do CPU hotplug events happen? Very rarely compared to _other_ valid preempting code, e.g. audio IRQs... And I don't see _any_ reason why CPU hotplug couldn't wait until a cpufreq request for a CPU is finished. However, the current CPU hotplug notifiers don't allow for this: they only notify the notifiers if the CPU is down, not before to avoid migrating to/switching frequency of the dying CPU.
Quoted 3 lines
Well interrupts would still preempt (side note, do any of these hardware programming have timing requirements? If so preempt might be a problem anyway) those scheduling preemption disabled sections.
Interrupts yes, but not interrupt handling threads [voluntary_preempt] or even "interrupt users" like a video player which requested more data... And concerning hardwre requirements: AFAIK there are only requirements of _minimum_ and not maximum delay between certain "steps" in the switching process.
Quoted 17 lines
Quoted 17 lines
A)
DOWN:
1.) notifier informs cpufreq that CPU is dying
2.) cpufreq grabs policy lock
3.) sets a flag or something so that future cpufreq-related calls for this
CPU will fail
4.) cpufreq release policy lock
5.) cpu is removed fromonline_map....UP:
1.) cpu is online
2.) notifier informs cpufreq that CPU is ONLINE
3.) cpufreq grabs policy lock
4.) clears the flag mentioned above
Quoted 7 lines
Quoted 3 lines
Problem: ->exit() and ->init() aren't called for the CPU going off- or online which means that changes in their cpufreq capabilities won't be noticed.
Well the major differences between A and B appear to be discovery of new capabilities on online processors. I wouldn't go that far as asynchronous multiprocessing systems aren't supported in general,
_yet_. AMD allows for different max freqs in their Opteron systems, the new time-of-day core will support different-freq CPUs soon, so this is indeed an issue.
Thanks,
Dominik