Skip to contentSkip to conversation

Subject: cell: add spu aware cpufreq governor

PATCH In review Christian Krafft 3 participants 10 messages Jan 18, 2008 – Apr 16, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

From: Christian Krafft <[email protected]>

This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.

This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).

arch/powerpc/platforms/cell/Kconfig +9 -0
Index: linux.git/arch/powerpc/platforms/cell/Kconfig
===================================================================
@@ -87,4 +87,13 @@ config CBE_CPUFREQ_PMI
 	  processor will not only be able to run at lower speed,
 	  but also at lower core voltage.
 
+config CBE_CPUFREQ_SPU_GOVERNOR
+	tristate "CBE frequency scaling based on SPU usage"
+	depends on CBE_CPUFREQ
+	default m
+	help
+	  This governor checks for spu usage to adjust the cpu frequency.
+	  If no spu is running on a given cpu, that cpu will be throttled to
+	  the minimal possible frequency.
+
 endmenu
arch/powerpc/platforms/cell/Makefile +1 -0
Index: linux.git/arch/powerpc/platforms/cell/Makefile
===================================================================
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM)			+= cbe_thermal
 obj-$(CONFIG_CBE_CPUFREQ_PMI)		+= cbe_cpufreq_pmi.o
 obj-$(CONFIG_CBE_CPUFREQ)		+= cbe-cpufreq.o
 cbe-cpufreq-y				+= cbe_cpufreq_pervasive.o cbe_cpufreq.o
+obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR)	+= cbe_spu_governor.o
 
 ifeq ($(CONFIG_SMP),y)
 obj-$(CONFIG_PPC_CELL_NATIVE)		+= smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +191 -0

191 lines not on this page · raw message

Signature · 13 lines
Mit freundlichen Gruessen,
kind regards,

Christian Krafft
IBM Systems & Technology Group,
Linux Kernel Development
IT Specialist


Vorsitzender des Aufsichtsrats:	Martin Jetter
Geschaeftsfuehrung:		Herbert Kircher
Sitz der Gesellschaft:		Boeblingen
Registriergericht:		Amtsgericht Stuttgart, HRB 243294

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq

#1 lore raw

arch/powerpc/platforms/cell/cbe_spu_governor.c line 32

Quoted 10 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+#include <linux/cpufreq.h>
+#include <linux/sched.h>
+#include <linux/timer.h>
+#include <asm/atomic.h>
+#include <asm/machdep.h>
+#include <asm/spu.h>
+
+#include "asm/cell-regs.h"
+
+#define DEBUG		1

You should have the '#define DEBUG' _before_ the #includes, otherwise pr_debug and friends don't do what you think they do.

Quoted 1 line

+static void spu_gov_init_work(struct spu_gov_info_struct *info);

Try to avoid forward declarations for static functions, instead reorder the functions in call order.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+static void spu_gov_timer(struct work_struct *work)

'timer' is a bad name for a work function, because it suggests you're working on a timer_list.

arch/powerpc/platforms/cell/cbe_spu_governor.c line 96

Quoted 7 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +7 -0
+static void spu_gov_init_work(struct spu_gov_info_struct *info)
+{
+	int delay = usecs_to_jiffies(info->poll_int * 1000);
+	delay -= jiffies % delay;
+	INIT_DELAYED_WORK_DEFERRABLE(&info->work, spu_gov_timer);
+	queue_delayed_work_on(info->policy->cpu, kspugov_wq, &info->work, delay);
+}

I think you shouldn't call INIT_DELAYED_WORK_DEFERRABLE() more than once. While I'm not sure if it's actually broken like you do it, it's certainly not how this interface was meant to be used.

What's the point of the computation, can't you just always queue the work to be run in info->poll_int miliseconds from the time you call it?

arch/powerpc/platforms/cell/cbe_spu_governor.c line 101

Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4 -0
+static void spu_gov_cancel_work(struct spu_gov_info_struct *info)
+{
+	cancel_delayed_work(&info->work);
+}

Doesn't that need to be cancel_delayed_work_sync() for rearming work queues?

arch/powerpc/platforms/cell/cbe_spu_governor.c line 168

Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4 -0
+static int __init spu_gov_init(void)
+{
+	if (!machine_is(cell))
+		return -ENODEV;

This seems wrong, why would you only allow that on cell platforms, but e.g. not celleb or even ps3 (assuming they had a cpufreq backend)? I think the only dependency is on spufs being loaded and that should work using module dependencies.

Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4 -0
+	kspugov_wq = create_workqueue("kspugov");
+	if (!kspugov_wq) {
+		printk(KERN_ERR "creation of kspugov failed\n");
+		return -EFAULT;

arch/powerpc/platforms/cell/cbe_spu_governor.c line 183

Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+	}
+
+	return cpufreq_register_governor(&spu_governor);
+}
+
+static void __exit spu_gov_exit(void)
+{
+	cpufreq_unregister_governor(&spu_governor);
+	destroy_workqueue(kspugov_wq);
+}

What happens when the work functions is called after unregistering?

	Arnd <><
#2 lore raw

Hi Arnd,

first of all, sorry for the late answer.
Here are my comments:

Quoted 14 lines · On Sat, 19 Jan 2008 22:38:44 +0100 Arnd Bergmann <[email protected]> wrote:

arch/powerpc/platforms/cell/cbe_spu_governor.c line 32

Quoted 10 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+#include <linux/cpufreq.h>
+#include <linux/sched.h>
+#include <linux/timer.h>
+#include <asm/atomic.h>
+#include <asm/machdep.h>
+#include <asm/spu.h>
+
+#include "asm/cell-regs.h"
+
+#define DEBUG		1

You should have the '#define DEBUG' _before_ the #includes, otherwise pr_debug and friends don't do what you think they do.

sure, DEBUG should have been removed anyway.

Quoted 4 lines
Quoted 1 line

+static void spu_gov_init_work(struct spu_gov_info_struct *info);

Try to avoid forward declarations for static functions, instead reorder the functions in call order.

Forward declaration was not a good idea at all, cause multiple calls to the init_work function seems a bit odd. Fixed with next version.

Quoted 4 lines
Quoted 1 line

+static void spu_gov_timer(struct work_struct *work)

'timer' is a bad name for a work function, because it suggests you're working on a timer_list.

this was a relic from a timer based version, renamed.

Quoted 11 lines
Quoted 7 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6 -0
+static void spu_gov_init_work(struct spu_gov_info_struct *info)
+{
+	int delay = usecs_to_jiffies(info->poll_int * 1000);
+	delay -= jiffies % delay;
+	INIT_DELAYED_WORK_DEFERRABLE(&info->work, spu_gov_timer);
+	queue_delayed_work_on(info->policy->cpu, kspugov_wq, &info->work,

delay); +}

I think you shouldn't call INIT_DELAYED_WORK_DEFERRABLE() more than once. While I'm not sure if it's actually broken like you do it, it's certainly not how this interface was meant to be used.

fixed with the removal of that odd forward declaration

Quoted 2 lines

What's the point of the computation, can't you just always queue the work to be run in info->poll_int miliseconds from the time you call it?

This code is from the ondemand governor. It should cause the timer to be triggered at the same jiffy. As our polling intervall is much higher I guess this makes no sense here -> removed.

It was also a stupid idea to use usecs_to_jiffies instead of msecs ;-)

Quoted 6 lines

arch/powerpc/platforms/cell/cbe_spu_governor.c line 101

Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4 -0
+static void spu_gov_cancel_work(struct spu_gov_info_struct *info)
+{
+	cancel_delayed_work(&info->work);
+}

Doesn't that need to be cancel_delayed_work_sync() for rearming work queues?

Jep.

Quoted 9 lines

arch/powerpc/platforms/cell/cbe_spu_governor.c line 168

Quoted 4 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +4 -0
+static int __init spu_gov_init(void)
+{
+	if (!machine_is(cell))
+		return -ENODEV;

This seems wrong, why would you only allow that on cell platforms, but e.g. not celleb or even ps3 (assuming they had a cpufreq backend)? I think the only dependency is on spufs being loaded and that should work using module dependencies.

Sure.
I added the dependency to CONFIG_SPUFS.
I also removed the dependency to CBE_CPUFREQ and replaced it by a select CBE_CPUFREQ.

Quoted 8 lines
Quoted 8 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +8 -0
+	kspugov_wq = create_workqueue("kspugov");
+	if (!kspugov_wq) {
+		printk(KERN_ERR "creation of kspugov failed\n");
+		return -EFAULT;
+	}
+
+	return cpufreq_register_governor(&spu_governor);
+}
Quoted 8 lines

arch/powerpc/platforms/cell/cbe_spu_governor.c line 183

Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6 -0
+
+static void __exit spu_gov_exit(void)
+{
+	cpufreq_unregister_governor(&spu_governor);
+	destroy_workqueue(kspugov_wq);
+}

What happens when the work functions is called after unregistering?

This can not happen, as module can only be unloaded if the spu_gov_govern got CPUFREQ_GOV_STOP for all cpus it has been started for. When getting CPUFREQ_GOV_STOP the cancel_delayed_work_SYNC function gets called. After spu_gov_govern processed the last CPUFREQ_GOV_STOP, no work function gets called. I added a BUG_ON with a comment to make that clear.

Thanks for the review,
Hope next version is clean ...

Signature · 13 lines
Mit freundlichen Gruessen,
kind regards,

Christian Krafft
IBM Systems & Technology Group,
Linux Kernel Development
IT Specialist


Vorsitzender des Aufsichtsrats:	Martin Jetter
Geschaeftsfuehrung:		Herbert Kircher
Sitz der Gesellschaft:		Boeblingen
Registriergericht:		Amtsgericht Stuttgart, HRB 243294

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq

#3 lore raw

PATCH 1/1 Resending: cell: add spu aware cpufreq governor

From: Christian Krafft <[email protected]>

This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.

This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).

arch/powerpc/platforms/cell/Kconfig +10 -0
Index: linux.git/arch/powerpc/platforms/cell/Kconfig
===================================================================
@@ -87,4 +87,14 @@ config CBE_CPUFREQ_PMI
 	  processor will not only be able to run at lower speed,
 	  but also at lower core voltage.
 
+config CBE_CPUFREQ_SPU_GOVERNOR
+	tristate "CBE frequency scaling based on SPU usage"
+	depends SPU_FS
+	select CBE_CPUFREQ
+	default m
+	help
+	  This governor checks for spu usage to adjust the cpu frequency.
+	  If no spu is running on a given cpu, that cpu will be throttled to
+	  the minimal possible frequency.
+
 endmenu
arch/powerpc/platforms/cell/Makefile +1 -0
Index: linux.git/arch/powerpc/platforms/cell/Makefile
===================================================================
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM)			+= cbe_thermal
 obj-$(CONFIG_CBE_CPUFREQ_PMI)		+= cbe_cpufreq_pmi.o
 obj-$(CONFIG_CBE_CPUFREQ)		+= cbe-cpufreq.o
 cbe-cpufreq-y				+= cbe_cpufreq_pervasive.o cbe_cpufreq.o
+obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR)	+= cbe_spu_governor.o
 
 ifeq ($(CONFIG_SMP),y)
 obj-$(CONFIG_PPC_CELL_NATIVE)		+= smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +199 -0

199 lines not on this page · raw message

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq

#4 lore raw

Re: [Cbe-oss-dev] [Patch] Resending: cell: add spu aware cpufreq governor

Sorry for my late reply, I lost track of this discussion when it happened during LCA and never got back to it. The patch looks good enough for 2.6.26, I think, but one thing still really bothers me:

Quoted 7 lines · Christian Krafft
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +7 -0
+/* parts of this function should go into spu scheduler */
+static int spu_gov_calc_load(struct spu_gov_info_struct *info)
+{
+       unsigned long active_tasks; /* fixed-point */
+       int cpu, load;
+
+       cpu = info->policy->cpu;

arch/powerpc/platforms/cell/cbe_spu_governor.c line 61

Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+       active_tasks = cbe_spu_info[cpu_to_node(cpu)].nr_active * FIXED_1;
+
+       /* this is also a bit too trivial
+        * actually we want the max load of all spu's belonging together */
+       CALC_LOAD(info->load, EXP_1, active_tasks);
+
+       load = (info->load + FIXED_1 / 200) >> FSHIFT;
+
+       return load;
+}

The nr_active variable in cbe_spu_info has a completely different meaning from nr_active() in the linux scheduler. AFAICS, basing your computation on this makes no sense whatsoever. Instead of looking at what threads are running on the SPUs, you look at what is loaded. Because of our lazy loading scheduler mechanism, a task occupying all SPUs at 10% of the time and otherwise calling nanosleep or futex_wait will still show up as 100% load here, so we don't lower the frequency although we should.

What's worse, a compute-intensive task running on half of the SPUs with the rest of them being idle shows up as 50% load and causes the frequency to be dropped to 50%, when it should run at full speed.

	Arnd <><
#5 lore raw

Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff

From: Christian Krafft <[email protected]>

This patch adds a cpufreq governor that takes the spu load into account. It's very similar to the ondemand governor, but not as complex. Instead of hacking spu load into the ondemand governor I'd like to see cpufreq accepting multiple governors per cpu in future. Don't know if this is the right way, but it would keep the governors simple.

This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).

This resent version fixes also corrects the depends clause in the Kconfig file.

---

arch/powerpc/platforms/cell/Kconfig +10 -0
Index: linux.git/arch/powerpc/platforms/cell/Kconfig
===================================================================
@@ -94,6 +94,16 @@ config CBE_CPUFREQ_PMI
 	  processor will not only be able to run at lower speed,
 	  but also at lower core voltage.
 
+config CBE_CPUFREQ_SPU_GOVERNOR
+	tristate "CBE frequency scaling based on SPU usage"
+	depends on SPU_FS
+	select CBE_CPUFREQ
+	default m
+	help
+	  This governor checks for spu usage to adjust the cpu frequency.
+	  If no spu is running on a given cpu, that cpu will be throttled to
+	  the minimal possible frequency.
+
 endmenu
 
 config OPROFILE_CELL
arch/powerpc/platforms/cell/Makefile +1 -0
Index: linux.git/arch/powerpc/platforms/cell/Makefile
===================================================================
@@ -7,6 +7,7 @@ obj-$(CONFIG_CBE_THERM)			+= cbe_thermal
 obj-$(CONFIG_CBE_CPUFREQ_PMI)		+= cbe_cpufreq_pmi.o
 obj-$(CONFIG_CBE_CPUFREQ)		+= cbe-cpufreq.o
 cbe-cpufreq-y				+= cbe_cpufreq_pervasive.o cbe_cpufreq.o
+obj-$(CONFIG_CBE_CPUFREQ_SPU_GOVERNOR)	+= cbe_spu_governor.o
 
 ifeq ($(CONFIG_SMP),y)
 obj-$(CONFIG_PPC_CELL_NATIVE)		+= smp.o
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +199 -0

199 lines not on this page · raw message

Signature · 13 lines
Mit freundlichen Gruessen,
kind regards,

Christian Krafft
IBM Systems & Technology Group,
Linux Kernel Development
IT Specialist


Vorsitzender des Aufsichtsrats:	Martin Jetter
Geschaeftsfuehrung:		Herbert Kircher
Sitz der Gesellschaft:		Boeblingen
Registriergericht:		Amtsgericht Stuttgart, HRB 243294

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq

#6 lore raw

Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff

Quoted 4 lines · Christian Krafft

This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).

So why should that be something to optimize for? If the SPUs are all idle, I would assume we'd be better off throwing the tasks off and putting them into low-power mode, rather than changing the frequency, right?

What's the point of your driver, really?

	Arnd <><
#7 lore raw

Re: [Cbe-oss-dev] [Patch] Resending: cell_add_spuaware_cpufreq_governor.diff

Quoted 6 lines · On Wed, 16 Apr 2008 14:08:25 +0200 Arnd Bergmann <[email protected]> wrote:
Quoted 4 lines · Christian Krafft

This patch is also missing a correct load calculation.
It works pretty well for spu's running at full time or idling, but not so well for mixed load (i.e. each spu running 50 percent of the time we would switch to fullspeed instead of half speed).

Quoted 7 lines

So why should that be something to optimize for? If the SPUs are all idle, I would assume we'd be better off throwing the tasks off and putting them into low-power mode, rather than changing the frequency, right?

What's the point of your driver, really?

	Arnd <><

What we have now is good enough to
- switch to full power if there are spus active (waiting or running) and
- switch to low power if not.

What we have now is not good enough to
- switch to low power if there are spus waiting, but none are running

As we cannot _yet_ differentiate between spus waiting and spus running means that in _some_ cases we will not save as much power as we could.

The point of submitting the driver is to get it working. Further improvements will follow of course.

Signature · 13 lines
Mit freundlichen Gruessen,
kind regards,

Christian Krafft
IBM Systems & Technology Group,
Linux Kernel Development
IT Specialist


Vorsitzender des Aufsichtsrats:	Martin Jetter
Geschaeftsfuehrung:		Herbert Kircher
Sitz der Gesellschaft:		Boeblingen
Registriergericht:		Amtsgericht Stuttgart, HRB 243294

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq

#8 lore raw
In reply to Christian Krafft
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6 -0
+/*
+ * module init and destoy
+ */
+
+static int __init spu_gov_init(void)
+{

arch/powerpc/platforms/cell/cbe_spu_governor.c line 176

Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+if (!machine_is(cell))
+return -ENODEV;
+
+kspugov_wq = create_workqueue("kspugov");
+if (!kspugov_wq) {
+printk(KERN_ERR "creation of kspugov failed\n");
+return -EFAULT;
+}
+
+return cpufreq_register_governor(&spu_governor);

Missing an error handling. This line should be something like:

    err = cpufreq_register_governor(&spu_governor);
    if (err) {
        destroy_workqueue(kspugov_wq);
        return err;
    }

    return 0;

It seems like this is copied from drivers/cpufreq/cpufreq_governor.c Althought this is not big problem, it would be nice to fix as well.

Quoted 1 line

+}

#9 lore raw

First of all, sorry for the delayed answer.

I had no hardware access (and no mail) last week,
I'll provide an updated patch very soon.

And thanks for your comments ;-)

Cheers,
Christian

Quoted 34 lines · On Mon, 21 Jan 2008 12:09:23 +0900 (JST) "Akinobu Mita" <[email protected]>…
Quoted 6 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +6 -0
+/*
+ * module init and destoy
+ */
+
+static int __init spu_gov_init(void)
+{

arch/powerpc/platforms/cell/cbe_spu_governor.c line 176

Quoted 10 lines
arch/powerpc/platforms/cell/cbe_spu_governor.c new file +10 -0
+if (!machine_is(cell))
+return -ENODEV;
+
+kspugov_wq = create_workqueue("kspugov");
+if (!kspugov_wq) {
+printk(KERN_ERR "creation of kspugov failed\n");
+return -EFAULT;
+}
+
+return cpufreq_register_governor(&spu_governor);

Missing an error handling. This line should be something like:

    err = cpufreq_register_governor(&spu_governor);
    if (err) {
        destroy_workqueue(kspugov_wq);
        return err;
    }

    return 0;

It seems like this is copied from drivers/cpufreq/cpufreq_governor.c Althought this is not big problem, it would be nice to fix as well.

Quoted 1 line

+}

Signature · 13 lines
Mit freundlichen Gruessen,
kind regards,

Christian Krafft
IBM Systems & Technology Group,
Linux Kernel Development
IT Specialist


Vorsitzender des Aufsichtsrats:	Martin Jetter
Geschaeftsfuehrung:		Herbert Kircher
Sitz der Gesellschaft:		Boeblingen
Registriergericht:		Amtsgericht Stuttgart, HRB 243294

_______________________________________________
Cpufreq mailing list
[email protected]
http://lists.linux.org.uk/mailman/listinfo/cpufreq