cpufreq: queue policy->update work to rt thread to reduce its schedule latency
PATCH 1/1 cpufreq: queue policy->update work to rt thread to reduce its schedule latency
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of policy->update work.
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the policy->update work to rt thread to reduce its schedule latency.
--- drivers/cpufreq/cpufreq.c | 24 ++++++++++++++++++------ include/linux/cpufreq.h | 4 +++- 2 files changed, 21 insertions(+), 7 deletions(-)
drivers/cpufreq/cpufreq.c +18 -6
@@ -1193,7 +1193,7 @@ void refresh_frequency_limits(struct cpufreq_policy *policy) }EXPORT_SYMBOL (refresh_frequency_limits); -static void handle_update (struct work_struct *work) +static void handle_update (struct kthread_work *work) {struct cpufreq_policy *policy = container_of(work, struct cpufreq_policy, update);
@@ -1209,7 +1209,7 @@ static int cpufreq_notifier_min(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_min); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }
@@ -1218,7 +1218,7 @@ static int cpufreq_notifier_max(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_max); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }
@@ -1301,15 +1301,25 @@ static struct cpufreq_policy *cpufreq_policy_alloc(unsigned int cpu) goto err_min_qos_notifier; } + policy->worker =kthread_create_worker_on_cpu (cpu,0 ,"policy_worker%d" , cpu); +if (IS_ERR (policy->worker)) { +dev_err (dev,"Failed to create policy_worker%d\n" , cpu); + goto err_max_qos_notifier; + } + +sched_set_fifo_low (policy->worker->task);INIT_LIST_HEAD (&policy->policy_list);init_rwsem (&policy->rwsem);spin_lock_init (&policy->transition_lock);init_waitqueue_head (&policy->transition_wait); -INIT_WORK (&policy->update, handle_update); +kthread_init_work (&policy->update, handle_update); policy->cpu = cpu;return policy; +err_max_qos_notifier: +freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MAX , + &policy->nb_max); err_min_qos_notifier:freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MIN , &policy->nb_min);
@@ -1353,7 +1363,9 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) &policy->nb_min);/* Cancel any pending policy->update work before freeing the policy. */ -cancel_work_sync (&policy->update); +kthread_cancel_work_sync (&policy->update); +if (policy->worker) +kthread_destroy_worker (policy->worker);if (policy->max_freq_req) {/*
@@ -1802,7 +1814,7 @@ static unsigned int cpufreq_verify_current_freq(struct cpufreq_policy *policy, bcpufreq_out_of_sync (policy, new_freq);if (update) -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update); }return new_freq;
include/linux/cpufreq.h +3 -1
@@ -20,6 +20,7 @@#include <linux/spinlock.h> #include <linux/sysfs.h> #include <linux/minmax.h> +#include <linux/kthread.h> /********************************************************************* * CPUFREQ INTERFACE *
@@ -77,8 +78,9 @@ struct cpufreq_policy {void *governor_data;char last_governor[CPUFREQ_NAME_LEN ];/* last governor used */ -struct work_struct update;/* if update_policy() needs to be +struct kthread_work update;/* if update_policy() needs to be * called, but you're in IRQ context */ +struct kthread_worker *worker;struct freq_constraints constraints;struct freq_qos_request *min_freq_req;
Signature · 1 line
2.17.1
Quoted 4 lines · Gaowei Pu
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of
policy->updatework.
What is the tail latency now after your change?
Quoted 4 lines
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the
policy->updatework to rt thread to reduce its schedule latency.
If my understanding is correct, kthread has a default nice value of 0 and is not a rt thread.
I think the gain you see is
your patch created a dedicated kthread work queue on CPU 0. The work from policy change no longer have to compete time with other requests coming from schedule_work().
If the policy change really needs to get ahead of other tasks, I think you need a dedicated workqueue with alloc_workqueue() using WQ_HIGHPRI flag.
Tim
Quoted 108 lines
Signed-off-by Gaowei Pu <[email protected]>--- drivers/cpufreq/cpufreq.c | 24 ++++++++++++++++++------ include/linux/cpufreq.h | 4 +++- 2 files changed, 21 insertions(+), 7 deletions(-)drivers/cpufreq/cpufreq.c +18
-6@@ -1193,7 +1193,7 @@ void refresh_frequency_limits(struct cpufreq_policy *policy) }EXPORT_SYMBOL (refresh_frequency_limits); -static void handle_update (struct work_struct *work) +static void handle_update (struct kthread_work *work) {struct cpufreq_policy *policy = container_of(work, struct cpufreq_policy, update);@@ -1209,7 +1209,7 @@ static int cpufreq_notifier_min(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_min); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1218,7 +1218,7 @@ static int cpufreq_notifier_max(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_max); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1301,15 +1301,25 @@ static struct cpufreq_policy *cpufreq_policy_alloc(unsigned int cpu) goto err_min_qos_notifier; } + policy->worker =kthread_create_worker_on_cpu (cpu,0 ,"policy_worker%d" , cpu); +if (IS_ERR (policy->worker)) { +dev_err (dev,"Failed to create policy_worker%d\n" , cpu); + goto err_max_qos_notifier; + } + +sched_set_fifo_low (policy->worker->task);INIT_LIST_HEAD (&policy->policy_list);init_rwsem (&policy->rwsem);spin_lock_init (&policy->transition_lock);init_waitqueue_head (&policy->transition_wait); -INIT_WORK (&policy->update, handle_update); +kthread_init_work (&policy->update, handle_update); policy->cpu = cpu;return policy; +err_max_qos_notifier: +freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MAX , + &policy->nb_max); err_min_qos_notifier:freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MIN , &policy->nb_min);@@ -1353,7 +1363,9 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) &policy->nb_min);/* Cancel any pending policy->update work before freeing the policy. */ -cancel_work_sync (&policy->update); +kthread_cancel_work_sync (&policy->update); +if (policy->worker) +kthread_destroy_worker (policy->worker);if (policy->max_freq_req) {/* @@ -1802,7 +1814,7 @@ static unsigned int cpufreq_verify_current_freq(struct cpufreq_policy *policy, bcpufreq_out_of_sync (policy, new_freq);if (update) -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update); }return new_freq;include/linux/cpufreq.h +3
-1@@ -20,6 +20,7 @@#include <linux/spinlock.h> #include <linux/sysfs.h> #include <linux/minmax.h> +#include <linux/kthread.h> /********************************************************************* * CPUFREQ INTERFACE * @@ -77,8 +78,9 @@ struct cpufreq_policy {void *governor_data;char last_governor[CPUFREQ_NAME_LEN ];/* last governor used */ -struct work_struct update;/* if update_policy() needs to be +struct kthread_work update;/* if update_policy() needs to be * called, but you're in IRQ context */ +struct kthread_worker *worker;struct freq_constraints constraints;struct freq_qos_request *min_freq_req;
Hi Tim,
Quoted 13 lines · Tim Chen
Quoted 4 lines · Gaowei Pu
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of
policy->updatework.What is the tail latency now after your change?
Quoted 4 lines
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the
policy->updatework to rt thread to reduce its schedule latency.
Quoted 7 lines
If my understanding is correct, kthread has a default nice value of 0 and is not a rt thread.
I think the gain you see is
your patch created a dedicated kthread work queue on CPU 0. The work from policy change no longer have to compete time with other requests coming fromschedule_work().
It's not just other requests coming from schedule_work(), also some normal cfs tasks running on the same cpu.
In order to not competing time with the above threads, i change the thread policy to rt and prio set to 98 to reduce the schedule latency.
Quoted 4 lines
If the policy change really needs to get ahead
of other tasks, I think you need a dedicated
workqueue withalloc_workqueue()usingWQ_HIGHPRIflag.I think the cpufreq boost or limit action should be trigger in time to meet
performance and power demands. An dedicated workqueue with highpri will be better but maybe not good enough because cfs pick or preempt policy is not purely based on thread nice value. So i think the final solution is rt thread and the policy change work deserves it :)
thanks.
Quoted 112 lines
Tim
Quoted 108 lines
Signed-off-by Gaowei Pu <[email protected]>--- drivers/cpufreq/cpufreq.c | 24 ++++++++++++++++++------ include/linux/cpufreq.h | 4 +++- 2 files changed, 21 insertions(+), 7 deletions(-)drivers/cpufreq/cpufreq.c +18
-6@@ -1193,7 +1193,7 @@ void refresh_frequency_limits(struct cpufreq_policy *policy) }EXPORT_SYMBOL (refresh_frequency_limits); -static void handle_update (struct work_struct *work) +static void handle_update (struct kthread_work *work) {struct cpufreq_policy *policy = container_of(work, struct cpufreq_policy, update);@@ -1209,7 +1209,7 @@ static int cpufreq_notifier_min(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_min); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1218,7 +1218,7 @@ static int cpufreq_notifier_max(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_max); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1301,15 +1301,25 @@ static struct cpufreq_policy *cpufreq_policy_alloc(unsigned int cpu) goto err_min_qos_notifier; } + policy->worker =kthread_create_worker_on_cpu (cpu,0 ,"policy_worker%d" , cpu); +if (IS_ERR (policy->worker)) { +dev_err (dev,"Failed to create policy_worker%d\n" , cpu); + goto err_max_qos_notifier; + } + +sched_set_fifo_low (policy->worker->task);INIT_LIST_HEAD (&policy->policy_list);init_rwsem (&policy->rwsem);spin_lock_init (&policy->transition_lock);init_waitqueue_head (&policy->transition_wait); -INIT_WORK (&policy->update, handle_update); +kthread_init_work (&policy->update, handle_update); policy->cpu = cpu;return policy; +err_max_qos_notifier: +freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MAX , + &policy->nb_max); err_min_qos_notifier:freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MIN , &policy->nb_min);@@ -1353,7 +1363,9 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) &policy->nb_min);/* Cancel any pending policy->update work before freeing the policy. */ -cancel_work_sync (&policy->update); +kthread_cancel_work_sync (&policy->update); +if (policy->worker) +kthread_destroy_worker (policy->worker);if (policy->max_freq_req) {/* @@ -1802,7 +1814,7 @@ static unsigned int cpufreq_verify_current_freq(struct cpufreq_policy *policy, bcpufreq_out_of_sync (policy, new_freq);if (update) -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update); }return new_freq;include/linux/cpufreq.h +3
-1@@ -20,6 +20,7 @@#include <linux/spinlock.h> #include <linux/sysfs.h> #include <linux/minmax.h> +#include <linux/kthread.h> /********************************************************************* * CPUFREQ INTERFACE * @@ -77,8 +78,9 @@ struct cpufreq_policy {void *governor_data;char last_governor[CPUFREQ_NAME_LEN ];/* last governor used */ -struct work_struct update;/* if update_policy() needs to be +struct kthread_work update;/* if update_policy() needs to be * called, but you're in IRQ context */ +struct kthread_worker *worker;struct freq_constraints constraints;struct freq_qos_request *min_freq_req;
Quoted 10 lines · Gaowei Pu
Hi Tim,
Quoted 7 lines · Tim Chen
Quoted 4 lines · Gaowei Pu
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of
policy->updatework.What is the tail latency now after your change?
sorry missed this.
the tail latency now is about within 50 microseconds after my change.
Quoted 142 lines
Quoted 6 lines
Quoted 5 lines
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the
policy->updatework to rt thread to reduce its schedule latency.Quoted 7 lines
If my understanding is correct, kthread has a default nice value of 0 and is not a rt thread.
I think the gain you see is
your patch created a dedicated kthread work queue on CPU 0. The work from policy change no longer have to compete time with other requests coming fromschedule_work().It's not just other requests coming from
schedule_work(), also some normal cfs tasks running on the same cpu.In order to not competing time with the above threads, i change the thread policy to rt and prio set to 98 to reduce the schedule latency.
Quoted 4 lines
If the policy change really needs to get ahead of other tasks, I think you need a dedicated workqueue with
alloc_workqueue()usingWQ_HIGHPRIflag.I think the cpufreq boost or limit action should be trigger in time to meet performance and power demands. An dedicated workqueue with highpri will be better but maybe not good enough because cfs pick or preempt policy is not purely based on thread nice value. So i think the final solution is rt thread and the policy change work deserves it :)
thanks.Quoted 112 lines
Tim
Quoted 108 lines
Signed-off-by Gaowei Pu <[email protected]>--- drivers/cpufreq/cpufreq.c | 24 ++++++++++++++++++------ include/linux/cpufreq.h | 4 +++- 2 files changed, 21 insertions(+), 7 deletions(-)drivers/cpufreq/cpufreq.c +18
-6@@ -1193,7 +1193,7 @@ void refresh_frequency_limits(struct cpufreq_policy *policy) }EXPORT_SYMBOL (refresh_frequency_limits); -static void handle_update (struct work_struct *work) +static void handle_update (struct kthread_work *work) {struct cpufreq_policy *policy = container_of(work, struct cpufreq_policy, update);@@ -1209,7 +1209,7 @@ static int cpufreq_notifier_min(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_min); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1218,7 +1218,7 @@ static int cpufreq_notifier_max(struct notifier_block *nb, unsigned long freq, { struct cpufreq_policy *policy = container_of(nb, struct cpufreq_policy, nb_max); -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update);return 0 ; }@@ -1301,15 +1301,25 @@ static struct cpufreq_policy *cpufreq_policy_alloc(unsigned int cpu) goto err_min_qos_notifier; } + policy->worker =kthread_create_worker_on_cpu (cpu,0 ,"policy_worker%d" , cpu); +if (IS_ERR (policy->worker)) { +dev_err (dev,"Failed to create policy_worker%d\n" , cpu); + goto err_max_qos_notifier; + } + +sched_set_fifo_low (policy->worker->task);INIT_LIST_HEAD (&policy->policy_list);init_rwsem (&policy->rwsem);spin_lock_init (&policy->transition_lock);init_waitqueue_head (&policy->transition_wait); -INIT_WORK (&policy->update, handle_update); +kthread_init_work (&policy->update, handle_update); policy->cpu = cpu;return policy; +err_max_qos_notifier: +freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MAX , + &policy->nb_max); err_min_qos_notifier:freq_qos_remove_notifier (&policy->constraints,FREQ_QOS_MIN , &policy->nb_min);@@ -1353,7 +1363,9 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) &policy->nb_min);/* Cancel any pending policy->update work before freeing the policy. */ -cancel_work_sync (&policy->update); +kthread_cancel_work_sync (&policy->update); +if (policy->worker) +kthread_destroy_worker (policy->worker);if (policy->max_freq_req) {/* @@ -1802,7 +1814,7 @@ static unsigned int cpufreq_verify_current_freq(struct cpufreq_policy *policy, bcpufreq_out_of_sync (policy, new_freq);if (update) -schedule_work (&policy->update); +kthread_queue_work (policy->worker, &policy->update); }return new_freq;include/linux/cpufreq.h +3
-1@@ -20,6 +20,7 @@#include <linux/spinlock.h> #include <linux/sysfs.h> #include <linux/minmax.h> +#include <linux/kthread.h> /********************************************************************* * CPUFREQ INTERFACE * @@ -77,8 +78,9 @@ struct cpufreq_policy {void *governor_data;char last_governor[CPUFREQ_NAME_LEN ];/* last governor used */ -struct work_struct update;/* if update_policy() needs to be +struct kthread_work update;/* if update_policy() needs to be * called, but you're in IRQ context */ +struct kthread_worker *worker;struct freq_constraints constraints;struct freq_qos_request *min_freq_req;
Rafael J. Wysocki
maintainerQuoted 20 lines · Gaowei Pu
Hi Tim,
Quoted 16 lines · Tim Chen
Quoted 4 lines · Gaowei Pu
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of
policy->updatework.What is the tail latency now after your change?
Quoted 4 lines
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the
policy->updatework to rt thread to reduce its schedule latency.If my understanding is correct, kthread has a default nice value of 0 and is not a rt thread.
Quoted 7 lines
Quoted 4 lines
I think the gain you see is
your patch created a dedicated kthread work queue on CPU 0. The work from policy change no longer have to compete time with other requests coming fromschedule_work().It's not just other requests coming from
schedule_work(), also some normal cfs tasks running on the same cpu.
Do you have any data to support this statement?
Quoted 2 lines
In order to not competing time with the above threads, i change the thread policy to rt and prio set to 98 to reduce the schedule latency.
By how much?
Quoted 10 lines
Quoted 4 lines
If the policy change really needs to get ahead of other tasks, I think you need a dedicated workqueue with
alloc_workqueue()usingWQ_HIGHPRIflag.I think the cpufreq boost or limit action should be trigger in time to meet performance and power demands. An dedicated workqueue with highpri will be better but maybe not good enough because cfs pick or preempt policy is not purely based on thread nice value. So i think the final solution is rt thread and the policy change work deserves it :)
The "I think" and "maybe" in the above paragraph are not particularly convincing.
Switching it over to use a dedicated workqueue would be a no-brainer as using dedicated workqueues is recommended anyway and if it measurably improves performance, that's for the better.
However, making it use a worker thread the way this patch does requires quite a clear demonstration that the above is not sufficient.
Thanks!
Hi Rafael J,
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Gaowei Pu
Hi Tim,
Quoted 20 lines · Tim Chen
Quoted 4 lines · Gaowei Pu
Currently we encountered a problem that the cpufreq boost latency is about 10 milliseconds or worse when we boost through cpufreq QOS request under high workload scenarios, while the boost latency mainly consumed by schedule latency of
policy->updatework.What is the tail latency now after your change?
Quoted 4 lines
We should ensure the low schedule latency of cpu frequency limits work to meet performance and power demands. so queue the
policy->updatework to rt thread to reduce its schedule latency.If my understanding is correct, kthread has a default nice value of 0 and is not a rt thread.
I think the gain you see is
your patch created a dedicated kthread work queue on CPU 0. The work from policy change no longer have to compete time with other requests coming fromschedule_work().
Quoted 4 lines
Quoted 2 lines
It's not just other requests coming from
schedule_work(), also some normal cfs tasks running on the same cpu.Do you have any data to support this statement?
Yes, i have some systraces in high workload android scenarios.
will send to you if needed.
Quoted 5 lines
Quoted 3 lines
In order to not competing time with the above threads, i change the thread policy to rt and prio set to 98 to reduce the schedule latency.
By how much?
the tail latency now is about within 50 microseconds after my change.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
If the policy change really needs to get ahead of other tasks, I think you need a dedicated workqueue with
alloc_workqueue()usingWQ_HIGHPRIflag.I think the cpufreq boost or limit action should be trigger in time to meet performance and power demands. An dedicated workqueue with highpri will be better but maybe not good enough because cfs pick or preempt policy is not purely based on thread nice value. So i think the final solution is rt thread and the policy change work deserves it :)
Quoted 6 lines
The "I think" and "maybe" in the above paragraph are not particularly convincing.
Switching it over to use a dedicated workqueue would be a no-brainer as using dedicated workqueues is recommended anyway and if it measurably improves performance, that's for the better.
Ok, thanks for your and Tim's advice.
I used the dedicated workqueue with highpri(tested and the sched latency is within 1ms) to replace the rt thread and will send v2 later.
Quoted 5 lines
However, making it use a worker thread the way this patch does requires quite a clear demonstration that the above is not sufficient.
Thanks!