Subject: cell: add spu aware cpufreq governor
Subsystems (2)
From: Christian Krafft <[email protected]>
This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.
This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).
arch/powerpc/platforms/cell/Kconfig +9 -0
@@ -87,4 +87,13 @@ config CBE_CPUFREQ_PMI processor will not only be able to run at lower speed, but also at lower core voltage. +config CBE_CPUFREQ_SPU_GOVERNOR + tristate "CBE frequency scaling based on SPU usage" + depends on CBE_CPUFREQ + default m + help + This governor checks for spu usage to adjust the cpu frequency. + If no spu is running on a given cpu, that cpu will be throttled to + the minimal possible frequency. + endmenu
arch/powerpc/platforms/cell/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM) += cbe_thermal obj-$(CONFIG_CBE_CPUFREQ_PMI) += cbe_cpufreq_pmi.o obj-$(CONFIG_CBE_CPUFREQ) += cbe-cpufreq.o cbe-cpufreq-y += cbe_cpufreq_pervasive.o cbe_cpufreq.o +obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR) += cbe_spu_governor.o ifeq ($(CONFIG_SMP),y) obj-$(CONFIG_PPC_CELL_NATIVE) += smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +191 -0
191 lines not on this page · raw message
Signature · 13 lines
Mit freundlichen Gruessen, kind regards, Christian Krafft IBM Systems & Technology Group, Linux Kernel Development IT Specialist Vorsitzender des Aufsichtsrats: Martin Jetter Geschaeftsfuehrung: Herbert Kircher Sitz der Gesellschaft: Boeblingen Registriergericht: Amtsgericht Stuttgart, HRB 243294
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq
arch/powerpc/platforms/cell/cbe_spu_governor.c line 32
Quoted 10 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+#include <linux/cpufreq.h> +#include <linux/sched.h> +#include <linux/timer.h> +#include <asm/atomic.h> +#include <asm/machdep.h> +#include <asm/spu.h> + +#include "asm/cell-regs.h" + +#define DEBUG 1
You should have the '#define DEBUG' _before_ the #includes, otherwise pr_debug and friends don't do what you think they do.
Quoted 1 line
+static void
spu_gov_init_work(struct spu_gov_info_struct *info);
Try to avoid forward declarations for static functions, instead reorder the functions in call order.
Review
Quoted 2 lines
(quoted diff) +2
-0+ +static void spu_gov_timer(struct work_struct *work)
'timer' is a bad name for a work function, because it suggests you're working on a timer_list.
arch/powerpc/platforms/cell/cbe_spu_governor.c line 96
Quoted 7 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +7
-0+static void spu_gov_init_work (struct spu_gov_info_struct *info) +{ +int delay =usecs_to_jiffies (info->poll_int *1000 ); + delay -= jiffies % delay; +INIT_DELAYED_WORK_DEFERRABLE (&info->work, spu_gov_timer); +queue_delayed_work_on (info->policy->cpu, kspugov_wq, &info->work, delay); +}
I think you shouldn't call INIT_DELAYED_WORK_DEFERRABLE() more than once. While I'm not sure if it's actually broken like you do it, it's certainly not how this interface was meant to be used.
What's the point of the computation, can't you just always queue the work to be run in info->poll_int miliseconds from the time you call it?
arch/powerpc/platforms/cell/cbe_spu_governor.c line 101
Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4
-0+static void spu_gov_cancel_work (struct spu_gov_info_struct *info) +{ +cancel_delayed_work (&info->work); +}
Doesn't that need to be cancel_delayed_work_sync() for rearming work queues?
arch/powerpc/platforms/cell/cbe_spu_governor.c line 168
Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4
-0+static int __init spu_gov_init (void ) +{ +if (!machine_is (cell)) +return -ENODEV ;
This seems wrong, why would you only allow that on cell platforms, but e.g. not celleb or even ps3 (assuming they had a cpufreq backend)? I think the only dependency is on spufs being loaded and that should work using module dependencies.
Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4
-0+ kspugov_wq =create_workqueue ("kspugov" ); +if (!kspugov_wq) { +printk (KERN_ERR "creation of kspugov failed\n" ); +return -EFAULT ;
arch/powerpc/platforms/cell/cbe_spu_governor.c line 183
Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+ } + +return cpufreq_register_governor (&spu_governor); +} + +static void __exit spu_gov_exit(void) +{ +cpufreq_unregister_governor (&spu_governor); +destroy_workqueue (kspugov_wq); +}
What happens when the work functions is called after unregistering?
Arnd <><
Hi Arnd,
first of all, sorry for the late answer.
Here are my comments:
Quoted 14 lines · On Sat, 19 Jan 2008 22:38:44 +0100 Arnd Bergmann <[email protected]> wrote:
arch/powerpc/platforms/cell/cbe_spu_governor.c line 32
Quoted 10 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+#include <linux/cpufreq.h> +#include <linux/sched.h> +#include <linux/timer.h> +#include <asm/atomic.h> +#include <asm/machdep.h> +#include <asm/spu.h> + +#include "asm/cell-regs.h" + +#define DEBUG 1You should have the '#define DEBUG' _before_ the #includes, otherwise
pr_debugand friends don't do what you think they do.
sure, DEBUG should have been removed anyway.
Quoted 4 lines
Quoted 1 line
+static void
spu_gov_init_work(struct spu_gov_info_struct *info);Try to avoid forward declarations for static functions, instead reorder the functions in call order.
Forward declaration was not a good idea at all, cause multiple calls to the init_work function seems a bit odd. Fixed with next version.
Quoted 4 lines
Quoted 1 line
+static void
spu_gov_timer(struct work_struct *work)'timer' is a bad name for a work function, because it suggests you're working on a
timer_list.
this was a relic from a timer based version, renamed.
Quoted 11 lines
Quoted 7 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6
-0+static void spu_gov_init_work (struct spu_gov_info_struct *info) +{ +int delay =usecs_to_jiffies (info->poll_int *1000 ); + delay -= jiffies % delay; +INIT_DELAYED_WORK_DEFERRABLE (&info->work, spu_gov_timer); + queue_delayed_work_on(info->policy->cpu, kspugov_wq, &info->work,delay); +}
I think you shouldn't call
INIT_DELAYED_WORK_DEFERRABLE()more than once. While I'm not sure if it's actually broken like you do it, it's certainly not how this interface was meant to be used.
fixed with the removal of that odd forward declaration
Quoted 2 lines
What's the point of the computation, can't you just always queue the work to be run in
info->poll_intmiliseconds from the time you call it?
This code is from the ondemand governor. It should cause the timer to be triggered at the same jiffy. As our polling intervall is much higher I guess this makes no sense here -> removed.
It was also a stupid idea to use usecs_to_jiffies instead of msecs ;-)
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c line 101
Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4
-0+static void spu_gov_cancel_work (struct spu_gov_info_struct *info) +{ +cancel_delayed_work (&info->work); +}Doesn't that need to be
cancel_delayed_work_sync()for rearming work queues?
Jep.
Quoted 9 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c line 168
Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4
-0+static int __init spu_gov_init (void ) +{ +if (!machine_is (cell)) +return -ENODEV ;This seems wrong, why would you only allow that on cell platforms, but e.g. not celleb or even ps3 (assuming they had a cpufreq backend)? I think the only dependency is on spufs being loaded and that should work using module dependencies.
Sure.
I added the dependency to CONFIG_SPUFS.
I also removed the dependency to CBE_CPUFREQ and replaced it by a select CBE_CPUFREQ.
Quoted 8 lines
Quoted 8 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +8
-0+ kspugov_wq =create_workqueue ("kspugov" ); +if (!kspugov_wq) { +printk (KERN_ERR "creation of kspugov failed\n" ); +return -EFAULT ; + } + +return cpufreq_register_governor (&spu_governor); +}
Quoted 8 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c line 183
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6
-0+ +static void __exit spu_gov_exit(void) +{ +cpufreq_unregister_governor (&spu_governor); +destroy_workqueue (kspugov_wq); +}What happens when the work functions is called after unregistering?
This can not happen, as module can only be unloaded if the spu_gov_govern got CPUFREQ_GOV_STOP for all cpus it has been started for. When getting CPUFREQ_GOV_STOP the cancel_delayed_work_SYNC function gets called. After spu_gov_govern processed the last CPUFREQ_GOV_STOP, no work function gets called. I added a BUG_ON with a comment to make that clear.
Thanks for the review,
Hope next version is clean ...
Signature · 13 lines
Mit freundlichen Gruessen, kind regards, Christian Krafft IBM Systems & Technology Group, Linux Kernel Development IT Specialist Vorsitzender des Aufsichtsrats: Martin Jetter Geschaeftsfuehrung: Herbert Kircher Sitz der Gesellschaft: Boeblingen Registriergericht: Amtsgericht Stuttgart, HRB 243294
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq
PATCH 1/1 Resending: cell: add spu aware cpufreq governor
From: Christian Krafft <[email protected]>
This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.
This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).
arch/powerpc/platforms/cell/Kconfig +10 -0
@@ -87,4 +87,14 @@ config CBE_CPUFREQ_PMI processor will not only be able to run at lower speed, but also at lower core voltage. +config CBE_CPUFREQ_SPU_GOVERNOR + tristate "CBE frequency scaling based on SPU usage" + depends SPU_FS + select CBE_CPUFREQ + default m + help + This governor checks for spu usage to adjust the cpu frequency. + If no spu is running on a given cpu, that cpu will be throttled to + the minimal possible frequency. + endmenu
arch/powerpc/platforms/cell/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM) += cbe_thermal obj-$(CONFIG_CBE_CPUFREQ_PMI) += cbe_cpufreq_pmi.o obj-$(CONFIG_CBE_CPUFREQ) += cbe-cpufreq.o cbe-cpufreq-y += cbe_cpufreq_pervasive.o cbe_cpufreq.o +obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR) += cbe_spu_governor.o ifeq ($(CONFIG_SMP),y) obj-$(CONFIG_PPC_CELL_NATIVE) += smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +199 -0
199 lines not on this page · raw message
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq
Re: [Cbe-oss-dev] [Patch] Resending: cell: add spu aware cpufreq governor
Sorry for my late reply, I lost track of this discussion when it happened during LCA and never got back to it. The patch looks good enough for 2.6.26, I think, but one thing still really bothers me:
Quoted 7 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +7
-0+/* parts of this function should go into spu scheduler */ +static int spu_gov_calc_load (struct spu_gov_info_struct *info) +{ + unsigned long active_tasks; /* fixed-point */ + int cpu, load; + + cpu = info->policy->cpu;
arch/powerpc/platforms/cell/cbe_spu_governor.c line 61
Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+ active_tasks = cbe_spu_info[cpu_to_node(cpu)].nr_active * FIXED_1; + + /* this is also a bit too trivial + * actually we want the max load of all spu's belonging together */ + CALC_LOAD(info->load, EXP_1, active_tasks); + + load = (info->load + FIXED_1 / 200) >> FSHIFT; + + return load; +}
The nr_active variable in cbe_spu_info has a completely different meaning from nr_active() in the linux scheduler. AFAICS, basing your computation on this makes no sense whatsoever. Instead of looking at what threads are running on the SPUs, you look at what is loaded. Because of our lazy loading scheduler mechanism, a task occupying all SPUs at 10% of the time and otherwise calling nanosleep or futex_wait will still show up as 100% load here, so we don't lower the frequency although we should.
What's worse, a compute-intensive task running on half of the SPUs with the rest of them being idle shows up as 50% load and causes the frequency to be dropped to 50%, when it should run at full speed.
Arnd <><
Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff
From: Christian Krafft <[email protected]>
This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.
This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).
This resent version fixes also corrects the depends clause in the Kconfig file.
---
arch/powerpc/platforms/cell/Kconfig +10 -0
@@ -94,6 +94,16 @@ config CBE_CPUFREQ_PMI processor will not only be able to run at lower speed, but also at lower core voltage. +config CBE_CPUFREQ_SPU_GOVERNOR + tristate "CBE frequency scaling based on SPU usage" + depends on SPU_FS + select CBE_CPUFREQ + default m + help + This governor checks for spu usage to adjust the cpu frequency. + If no spu is running on a given cpu, that cpu will be throttled to + the minimal possible frequency. + endmenu config OPROFILE_CELL
arch/powerpc/platforms/cell/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM) += cbe_thermal obj-$(CONFIG_CBE_CPUFREQ_PMI) += cbe_cpufreq_pmi.o obj-$(CONFIG_CBE_CPUFREQ) += cbe-cpufreq.o cbe-cpufreq-y += cbe_cpufreq_pervasive.o cbe_cpufreq.o +obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR) += cbe_spu_governor.o ifeq ($(CONFIG_SMP),y) obj-$(CONFIG_PPC_CELL_NATIVE) += smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +199 -0
199 lines not on this page · raw message
Signature · 13 lines
Mit freundlichen Gruessen, kind regards, Christian Krafft IBM Systems & Technology Group, Linux Kernel Development IT Specialist Vorsitzender des Aufsichtsrats: Martin Jetter Geschaeftsfuehrung: Herbert Kircher Sitz der Gesellschaft: Boeblingen Registriergericht: Amtsgericht Stuttgart, HRB 243294
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq
Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff
Quoted 4 lines · Christian Krafft
This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).
So why should that be something to optimize for? If the SPUs are all idle, I would assume we'd be better off throwing the tasks off and putting them into low-power mode, rather than changing the frequency, right?
What's the point of your driver, really?
Arnd <><
Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff
Quoted 6 lines · On Wed, 16 Apr 2008 14:08:25 +0200 Arnd Bergmann <[email protected]> wrote:
Quoted 4 lines · Christian Krafft
This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).
Quoted 7 lines
So why should that be something to optimize for? If the SPUs are all idle, I would assume we'd be better off throwing the tasks off and putting them into low-power mode, rather than changing the frequency, right?
What's the point of your driver, really?
Arnd <><
What we have now is good enough to
- switch to full power if there are spus active (waiting or running) and
- switch to low power if not.
What we have now is not good enough to
- switch to low power if there are spus waiting, but none are running
As we cannot _yet_ differentiate between spus waiting and spus running means that in _some_ cases we will not save as much power as we could.
The point of submitting the driver is to get it working. Further improvements will follow of course.
Signature · 13 lines
Mit freundlichen Gruessen, kind regards, Christian Krafft IBM Systems & Technology Group, Linux Kernel Development IT Specialist Vorsitzender des Aufsichtsrats: Martin Jetter Geschaeftsfuehrung: Herbert Kircher Sitz der Gesellschaft: Boeblingen Registriergericht: Amtsgericht Stuttgart, HRB 243294
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6
-0+/* +* module init and destoy +*/ + +static int __init spu_gov_init (void ) +{
arch/powerpc/platforms/cell/cbe_spu_governor.c line 176
Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+if (!machine_is (cell)) +return -ENODEV ; + +kspugov_wq =create_workqueue ("kspugov" ); +if (!kspugov_wq) { +printk (KERN_ERR "creation of kspugov failed\n" ); +return -EFAULT ; +} + +return cpufreq_register_governor (&spu_governor);
Missing an error handling. This line should be something like:
err = cpufreq_register_governor(&spu_governor);
if (err) {
destroy_workqueue(kspugov_wq);
return err;
}
return 0;It seems like this is copied from drivers/cpufreq/cpufreq_governor.c Althought this is not big problem, it would be nice to fix as well.
Quoted 1 line
+}
First of all, sorry for the delayed answer.
I had no hardware access (and no mail) last week,
I'll provide an updated patch very soon.
And thanks for your comments ;-)
Cheers,
Christian
Quoted 34 lines · On Mon, 21 Jan 2008 12:09:23 +0900 (JST) "Akinobu Mita" <[email protected]>…
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6
-0+/* +* module init and destoy +*/ + +static int __init spu_gov_init (void ) +{arch/powerpc/platforms/cell/cbe_spu_governor.c line 176
Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10
-0+if (!machine_is (cell)) +return -ENODEV ; + +kspugov_wq =create_workqueue ("kspugov" ); +if (!kspugov_wq) { +printk (KERN_ERR "creation of kspugov failed\n" ); +return -EFAULT ; +} + +return cpufreq_register_governor (&spu_governor);Missing an error handling. This line should be something like:
err = cpufreq_register_governor(&spu_governor); if (err) { destroy_workqueue(kspugov_wq); return err; } return 0;It seems like this is copied from
drivers/cpufreq/cpufreq_governor.cAlthought this is not big problem, it would be nice to fix as well.Quoted 1 line
+}
Signature · 13 lines
Mit freundlichen Gruessen, kind regards, Christian Krafft IBM Systems & Technology Group, Linux Kernel Development IT Specialist Vorsitzender des Aufsichtsrats: Martin Jetter Geschaeftsfuehrung: Herbert Kircher Sitz der Gesellschaft: Boeblingen Registriergericht: Amtsgericht Stuttgart, HRB 243294
_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq