Skip to contentSkip to conversation

TurboSched: A scheduler for sustaining Turbo Frequencies for longer durations

RFC In review Parth Shah 3 participants 14 messages Jan 21, 2020 – Jan 27, 2020

Subsystems (2)
lore
Share & tools
lore

Parth Shah

author
#0 lore raw

v6 cover TurboSched: A scheduler for sustaining Turbo Frequencies for longer durations

This is the 5th version of the patch series to sustain Turbo frequencies for longer durations.

The previous versions can be found here:
v5: https://lkml.org/lkml/2019/10/7/118
v4: https://lkml.org/lkml/2019/7/25/296
v3: https://lkml.org/lkml/2019/6/25/25
v2: https://lkml.org/lkml/2019/5/15/1258
v1: https://lwn.net/Articles/783959/

The changes in this versions are:
v5 -> v6:
- Addressed comments from Vincent Guittot and Hillf Danton
- Re based the series on the top of latency_nice patch series defined at
https://lkml.org/lkml/2020/1/16/319. This allows [1] to use the latency_nice framework for small background tasks classification from the userspace.
ToDo:
- Add Documentation for TurboSched including possible regression as per the comment from Pavel Machek

v4 -> v5:
- Remove Core capacity calculation for finding non-idle core
- Use idle_cpu() and cpu_overutilized() to find the core for task packing
- This changes functionality a bit. Updated new results for POWER9 system
- Re-named ambiguous naming "jitter" to "small background" tasks

v3 -> v4:
- Based on Patrick Bellasi's comments, removed the use of UCLAMP based mechanism to classify tasks as jitter
- Added support to sched_setattr to mark the task as jitter by adding a new flag to the existing task_struct->flags attribute. This is decided to not have any new variable inside task_struct and thus get rid of size bloating.
- No functional changes

v2 -> v3:
- Added a new attribute in task_struct to allow per task jitter classification so that scheduler can use this as request to change wakeup path for task packing
- Use syscall for jitter classification, removed cgroup based task classification
- Use mutex over spinlock to get rid of task sleeping problem
- Changed _Bool->int everywhere
- Split few patches to have arch specific code separate from core scheduler code

v1 -> v2:
- No CPU bound tasks' classification, only jitter tasks are classified from the cpu cgroup controller
- Use of Spinlock rather than mutex to count number of jitters in the system classified from cgroup
- Architecture specific implementation of Core capacity multiplication factor changes dynamically based on the number of active threads in the core
- Selection of non idle core in the system is bounded by DIE domain
- Use of UCLAMP mechanism to classify jitter tasks
- Removed "highutil_cpu_mask", and rather uses sd for DIE domain to find better fit

Abstract
========

The modern servers allows multiple cores to run at range of frequencies higher than rated range of frequencies. But the power budget of the system inhibits sustaining these higher frequencies for longer durations.

However when certain cores are put to idle states, the power can be effectively channelled to other busy cores, allowing them to sustain the higher frequency.

One way to achieve this is to pack tasks onto fewer cores keeping others idle, but it may lead to performance penalty for such tasks and sustaining higher frequencies proves to be of no benefit. But if one can identify unimportant low utilization tasks which can be packed on the already active cores then waking up of new cores can be avoided. Such tasks are short and/or bursty "background tasks" and waking up new core is expensive for such case.

Current CFS algorithm in kernel scheduler is performance oriented and hence tries to assign any idle CPU first for the waking up of new tasks. This policy is perfect for major categories of the workload, but for background tasks, one can save energy by packing them onto the active cores and allow those cores to run at higher frequencies.

These patch-set tunes the task wake up logic in scheduler to pack exclusively classified background tasks onto busy cores. The work involves the such tasks classifications by using syscall based mechanisms.

In brief, if we can pack such small background tasks on busy cores then we can save power by keeping other cores idle and allow busier cores to run at turbo frequencies, patch-set tries to meet this solution in simplest manner by only packing tasks with latency_nice==19 and util <= 12.5%.

Implementation
==============

These patches uses latency_nice [3] syscall based mechanism to classify the tasks as small background noises. The task wakeup logic uses this information to pack such tasks onto cores which are already running busy with CPU intensive tasks. The task packing is done at select_task_rq_fair only so that in case of wrong decision load balancer may pull the classified background tasks for maximizing performance.

We define a core to be non-idle if any CPU has >12.5% utilization and not more than 1 CPU is overutilized (>80% utilization); the background tasks are packed over these cores using First-fit approach.

The value 12.5% utilization indicates the CPU is sufficiently busy to not go to deeper IDLE-states (target_residency >= 10ms) and tasks can be packed here.

To demonstrate/benchmark, patches uses turbo_bench, a synthetic workload generator [2].

Following snippet demonstrates the use of TurboSched feature:

```
i=8; ./turbo_bench -t 30 -h $i -n $((i*2)) -j
```

This spawns 2*i total threads: of which i-CPU bound and i-low util threads.

Current implementation uses only small background classified tasks to be packed on the first busy cores, but can be further optimized by getting userspace input of important tasks and keeping track of such tasks. This leads to optimized searching of non idle cores and also more accurate as userspace hints are safer than auto classified busy cores/tasks.

Result
======

The patch-set proves to be useful for the system and the workload where frequency boost is found to be useful than packing tasks into cores. IBM POWER 9 system shows the frequency benefit can be up to 18% which can be translated to the maximum workload benefit up to 14%.

(higher is better)

Frequency benefit of TurboSched w.r.t. CFS

   +-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+
20 +-+ + + +  + + + + + + + +  + + + + + + + +  + + + + + + + +  + + + +-+
   |                    **                Frequency benefit in %         |
   |                    **                                               |
15 +-+                  **                                             +-+
   |              ****  **  **                                           |
   |            * ****  ******                                           |
10 +-+          * ****  ******                                         +-+
   |            * ****  ******                                           |
   |          * * ************   *                                       |
 5 +-+        * * ************ * *   **                                +-+
   |       ** * * ************ * *   ****                                |
 0 +-******** * * ************ * * ************ * * * ********** * * * **+
   |   **                                                                |
   |                                                                     |
-5 +-+                                                                 +-+
   | + + + +  + + + + + + + +  + + + + + + + +  + + + + + + + +  + + + + |
   +-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+
     0 1 2 3  4 5 6 7 8 91011 1213141516171819 2021222324252627 28293031  
                           No. of workload threads                        

Performance benefit of TurboSched w.r.t. CFS
20 +-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+

   | + + + +  + + + + + + + +  + + + + + + + +  + + + + + + + +  + + + + |
   |                                    Performance benefit in %         |
15 +-+                  **                                             +-+
   |                    **                                               |
   |                    ******                                           |
10 +-+                  ******                                         +-+
   |                **********                                           |
   |              ************                                           |
 5 +-+            ************   *     **                              +-+
   |              ************   *   ****                                |
   |            * ************ * *   ******  **                          |
 0 +-******** * * ************ * * ************ * * * ********** * * * **+
   |                                       **             **     *       |
   |                                                                     |
-5 +-+                                                                 +-+
   | + + + +  + + + + + + + +  + + + + + + + +  + + + + + + + +  + + + + |
   +-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+-+-+-+--+-+-+-+-+
     0 1 2 3  4 5 6 7 8 91011 1213141516171819 2021222324252627 28293031  
                           No. of workload threads                        

These numbers are w.r.t. turbo_bench.c multi-threaded test benchmark which can create two kinds of tasks: CPU bound (High Utilization) and Background (Low Utilization). N in X-axis represents N-CPU bound and N-background tasks spawned. The performance (Operations per Seconds) graph indicates the benefit with TurboSched can be upto 14% compared to the CFS task placement strategy for such background classified tasks.

Series organization
==============
- Patches 1-2: Small background tasks classification using syscall
- Patch 3 : Tune CFS task wakeup logic to pack tasks onto busy cores
- Patches 4-5: Change non-idle core search domain to LLC by default and provide arch hooks to change to NUMA for powerpc.

Series can be applied on top of latency_nice attribute introduction patches [3].

References
==========
[1]. Usecases for the per-task latency-nice attribute,

     https://lkml.org/lkml/2019/9/30/215
[2]. Test Benchmark: turbobench,
     https://github.com/parthsl/tools/blob/master/benchmarks/turbo_bench.c
[3]. Introduce per-task latency_nice for scheduler hints,
     https://lkml.org/lkml/2020/1/16/319

Parth Shah (5):
sched: Introduce switch to enable TurboSched for task packing sched/core: Update turbo_sched count only when required sched/fair: Tune task wake-up logic to pack small background tasks on
fewer cores
sched/fair: Provide arch hook to find domain for non idle core search
scan
powerpc: Set turbo domain to NUMA node for task packing

 arch/powerpc/include/asm/topology.h |  3 +
 arch/powerpc/kernel/smp.c           |  7 +++
 kernel/sched/core.c                 | 37 +++++++++++
 kernel/sched/fair.c                 | 95 ++++++++++++++++++++++++++++-
 kernel/sched/sched.h                | 15 +++++
 5 files changed, 156 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.2
author
#1 lore raw

RFC v6 1/5 sched: Introduce switch to enable TurboSched for task packing

Create a static key which allows to enable or disable TurboSched feature at runtime.

This key is added in order to enable the TurboSched feature only when required. This helps in optimizing the scheduler fast-path when the TurboSched feature is disabled.

Also provide get/put methods to keep track of the tasks using the TurboSched feature and also refcount classified background tasks. This allows to enable the feature on setting first task classified as background noise, similarly disable the feature on unsetting of such last task.

---
 kernel/sched/core.c  | 25 +++++++++++++++++++++++++
 kernel/sched/sched.h | 12 ++++++++++++
 2 files changed, 37 insertions(+)
kernel/sched/core.c +25 -0
index a9e5d157b1a5..dfbb52d66b29 100644
@@ -73,6 +73,31 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 
+#ifdef CONFIG_SCHED_SMT
+DEFINE_STATIC_KEY_FALSE(__turbo_sched_enabled);
+static DEFINE_MUTEX(turbo_sched_lock);
+static int turbo_sched_count;
+
+void turbo_sched_get(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!turbo_sched_count++)
+		static_branch_enable(&__turbo_sched_enabled);
+	mutex_unlock(&turbo_sched_lock);
+}
+
+void turbo_sched_put(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!--turbo_sched_count)
+		static_branch_disable(&__turbo_sched_enabled);
+	mutex_unlock(&turbo_sched_lock);
+}
+#else
+void turbo_sched_get(void) { return ; }
+void turbo_sched_get(void) { return ; }
+#endif
+
 /*
  * __task_rq_lock - lock the rq @p resides on.
  */
kernel/sched/sched.h +12 -0
index edae9277e48d..f841297b7d56 100644
@@ -2497,3 +2497,15 @@ static inline void membarrier_switch_mm(struct rq *rq,
 {
 }
 #endif
+
+void turbo_sched_get(void);
+void turbo_sched_put(void);
+
+#ifdef CONFIG_SCHED_SMT
+DECLARE_STATIC_KEY_FALSE(__turbo_sched_enabled);
+
+static inline bool is_turbosched_enabled(void)
+{
+	return static_branch_unlikely(&__turbo_sched_enabled);
+}
+#endif
Signature · 1 line
2.17.2
#2 lore raw

Re: [RFC v6 1/5] sched: Introduce switch to enable TurboSched for task packing

Quoted 26 lines · Parth Shah

Create a static key which allows to enable or disable TurboSched feature at runtime.

This key is added in order to enable the TurboSched feature only when required. This helps in optimizing the scheduler fast-path when the TurboSched feature is disabled.

Also provide get/put methods to keep track of the tasks using the TurboSched feature and also refcount classified background tasks. This allows to enable the feature on setting first task classified as background noise, similarly disable the feature on unsetting of such last task.

---
 kernel/sched/core.c  | 25 +++++++++++++++++++++++++
 kernel/sched/sched.h | 12 ++++++++++++
 2 files changed, 37 insertions(+)
kernel/sched/core.c +0 -0
index a9e5d157b1a5..dfbb52d66b29 100644
@@ -73,6 +73,31 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 

kernel/sched/core.c __read_mostly int scheduler_running; · line 85

Quoted 10 lines
kernel/sched/core.c +10 -0
+#ifdef CONFIG_SCHED_SMT
+DEFINE_STATIC_KEY_FALSE(__turbo_sched_enabled);
+static DEFINE_MUTEX(turbo_sched_lock);
+static int turbo_sched_count;
+
+void turbo_sched_get(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!turbo_sched_count++)
+		static_branch_enable(&__turbo_sched_enabled);

If you use static_branch_inc(&__turbo_sched_enabled) and static_branch_dec(&__turbo_sched_enabled), you don't have to define turbo_sched_count. And turbo_sched_lock is also unnecessary as static_branch_inc/dec are atomic.

kernel/sched/core.c __read_mostly int scheduler_running; · line 98

Quoted 13 lines
kernel/sched/core.c +13 -0
+	mutex_unlock(&turbo_sched_lock);
+}
+
+void turbo_sched_put(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!--turbo_sched_count)
+		static_branch_disable(&__turbo_sched_enabled);
+	mutex_unlock(&turbo_sched_lock);
+}
+#else
+void turbo_sched_get(void) { return ; }
+void turbo_sched_get(void) { return ; }

Double definition of turbo_sched_get.
You probably meant turbo_sched_put in the second definition.

Tim

author
#3 lore raw

Re: [RFC v6 1/5] sched: Introduce switch to enable TurboSched for task packing

Quoted 27 lines · Tim Chen
Quoted 26 lines · Parth Shah

Create a static key which allows to enable or disable TurboSched feature at runtime.

This key is added in order to enable the TurboSched feature only when required. This helps in optimizing the scheduler fast-path when the TurboSched feature is disabled.

Also provide get/put methods to keep track of the tasks using the TurboSched feature and also refcount classified background tasks. This allows to enable the feature on setting first task classified as background noise, similarly disable the feature on unsetting of such last task.

---
 kernel/sched/core.c  | 25 +++++++++++++++++++++++++
 kernel/sched/sched.h | 12 ++++++++++++
 2 files changed, 37 insertions(+)
kernel/sched/core.c +0 -0
index a9e5d157b1a5..dfbb52d66b29 100644
@@ -73,6 +73,31 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 
Quoted 16 lines

kernel/sched/core.c __read_mostly int scheduler_running; · line 85

Quoted 10 lines
kernel/sched/core.c +10 -0
+#ifdef CONFIG_SCHED_SMT
+DEFINE_STATIC_KEY_FALSE(__turbo_sched_enabled);
+static DEFINE_MUTEX(turbo_sched_lock);
+static int turbo_sched_count;
+
+void turbo_sched_get(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!turbo_sched_count++)
+		static_branch_enable(&__turbo_sched_enabled);

If you use static_branch_inc(&__turbo_sched_enabled) and static_branch_dec(&__turbo_sched_enabled), you don't have to define turbo_sched_count. And turbo_sched_lock is also unnecessary as static_branch_inc/dec are atomic.

That's a good suggestion. I will make those changes in the next version.

Quoted 8 lines
Quoted 8 lines
kernel/sched/core.c +8 -0
+	mutex_unlock(&turbo_sched_lock);
+}
+
+void turbo_sched_put(void)
+{
+	mutex_lock(&turbo_sched_lock);
+	if (!--turbo_sched_count)
+		static_branch_disable(&__turbo_sched_enabled);
Quoted 8 lines

kernel/sched/core.c __read_mostly int scheduler_running; · line 98

Quoted 5 lines
kernel/sched/core.c +5 -0
+	mutex_unlock(&turbo_sched_lock);
+}
+#else
+void turbo_sched_get(void) { return ; }
+void turbo_sched_get(void) { return ; }

Double definition of turbo_sched_get.
You probably meant turbo_sched_put in the second definition.

yes, my bad. I meant turbo_sched_put() instead.

Thanks,
Parth

Quoted 3 lines

Tim

#4 lore raw

Re: [RFC v6 1/5] sched: Introduce switch to enable TurboSched for task packing

In reply to Parth Shah
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Parth,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on powerpc/next tip/auto-latest linux/master linus/master v5.5-rc7 next-20200123] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Parth-Shah/TurboSched-A-scheduler-for-sustaining-Turbo-Frequencies-for-longer-durations/20200124-041428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git afa70d941f663c69c9a64ec1021bbcfa82f0e54a
config: xtensa-common_defconfig (attached as .config)
compiler: xtensa-linux-gcc (GCC) 7.5.0

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=xtensa

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

>> kernel/sched/core.c:98:6: error: redefinition of 'turbo_sched_get'
    void turbo_sched_get(void) { return ; }
         ^~~~~~~~~~~~~~~
   kernel/sched/core.c:97:6: note: previous definition of 'turbo_sched_get' was here
    void turbo_sched_get(void) { return ; }
         ^~~~~~~~~~~~~~~

vim +/turbo_sched_get +98 kernel/sched/core.c

    88	
    89	void turbo_sched_put(void)
    90	{
    91		mutex_lock(&turbo_sched_lock);
    92		if (!--turbo_sched_count)
    93			static_branch_disable(&__turbo_sched_enabled);
    94		mutex_unlock(&turbo_sched_lock);
    95	}
    96	#else
    97	void turbo_sched_get(void) { return ; }
  > 98	void turbo_sched_get(void) { return ; }
    99	#endif

100

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

author
#5 lore raw

RFC v6 2/5 sched/core: Update turbo_sched count only when required

In reply to Parth Shah

Use the get/put methods to add/remove the use of TurboSched support, such that the feature is turned on only in the presence of atleast one classified small bckground task.

---
 kernel/sched/core.c  | 9 +++++++++
 kernel/sched/sched.h | 3 +++
 2 files changed, 12 insertions(+)
kernel/sched/core.c +9 -0
index dfbb52d66b29..629c2589d727 100644
@@ -3272,6 +3272,9 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 		mmdrop(mm);
 	}
 	if (unlikely(prev_state == TASK_DEAD)) {
+		if (unlikely(is_bg_task(prev)))
+			turbo_sched_put();
+
 		if (prev->sched_class->task_dead)
 			prev->sched_class->task_dead(prev);
 
@@ -4800,6 +4803,8 @@ static int __sched_setscheduler(struct task_struct *p,
 	int reset_on_fork;
 	int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 	struct rq *rq;
+	bool attr_leniency = bgtask_latency(attr->sched_latency_nice);
+
 
 	/* The pi code expects interrupts enabled */
 	BUG_ON(pi && in_interrupt());
@@ -5024,6 +5029,10 @@ static int __sched_setscheduler(struct task_struct *p,
 
 	prev_class = p->sched_class;
 
+	/* Refcount tasks classified as a small background task */
+	if (is_bg_task(p) != attr_leniency)
+		attr_leniency ? turbo_sched_get() : turbo_sched_put();
+
 	__setscheduler(rq, p, attr, pi);
 	__setscheduler_uclamp(p, attr);
 
kernel/sched/sched.h +3 -0
index f841297b7d56..0a00e16e033a 100644
@@ -2498,6 +2498,9 @@ static inline void membarrier_switch_mm(struct rq *rq,
 }
 #endif
 
+#define bgtask_latency(lat)	((lat) == MAX_LATENCY_NICE)
+#define is_bg_task(p)		(bgtask_latency((p)->latency_nice))
+
 void turbo_sched_get(void);
 void turbo_sched_put(void);
 
Signature · 1 line
2.17.2
#6 lore raw

Re: [RFC v6 2/5] sched/core: Update turbo_sched count only when required

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Parth,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on powerpc/next tip/auto-latest linux/master linus/master v5.5-rc7 next-20200123] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Parth-Shah/TurboSched-A-scheduler-for-sustaining-Turbo-Frequencies-for-longer-durations/20200124-041428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git afa70d941f663c69c9a64ec1021bbcfa82f0e54a
config: i386-randconfig-c002-20200124 (attached as .config)
compiler: gcc-7 (Debian 7.5.0-3) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

In file included from arch/x86/include/asm/current.h:5:0,
from include/linux/sched.h:12,
from kernel/sched/sched.h:5,
from kernel/sched/core.c:9:
kernel/sched/core.c: In function 'finish_task_switch':
kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?

    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:33:34: note: in definition of macro '__branch_check__'
       ______r = __builtin_expect(!!(x), expect); \
                                     ^
>> kernel/sched/core.c:3272:7: note: in expansion of macro 'unlikely'
      if (unlikely(is_bg_task(prev)))
          ^~~~~~~~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_OPTINSN_SIZE'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:33:34: note: in definition of macro '__branch_check__'
       ______r = __builtin_expect(!!(x), expect); \
                                     ^
>> kernel/sched/core.c:3272:7: note: in expansion of macro 'unlikely'
      if (unlikely(is_bg_task(prev)))
          ^~~~~~~~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: note: each undeclared identifier is reported only once for each function it appears in
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:33:34: note: in definition of macro '__branch_check__'
       ______r = __builtin_expect(!!(x), expect); \
                                     ^
>> kernel/sched/core.c:3272:7: note: in expansion of macro 'unlikely'
      if (unlikely(is_bg_task(prev)))
          ^~~~~~~~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:35:19: note: in definition of macro '__branch_check__'
              expect, is_constant); \
                      ^~~~~~~~~~~
>> kernel/sched/core.c:3272:7: note: in expansion of macro 'unlikely'
      if (unlikely(is_bg_task(prev)))
          ^~~~~~~~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   In file included from kernel/sched/core.c:9:0:
   kernel/sched/core.c: In function '__sched_setscheduler':
   kernel/sched/core.c:4801:44: error: 'const struct sched_attr' has no member named 'sched_latency_nice'; did you mean 'sched_nice'?
     bool attr_leniency = bgtask_latency(attr->sched_latency_nice);
                                               ^
   kernel/sched/sched.h:2483:31: note: in definition of macro 'bgtask_latency'
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                  ^~~
   kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_OPTINSN_SIZE'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   kernel/sched/core.c:4801:23: note: in expansion of macro 'bgtask_latency'
     bool attr_leniency = bgtask_latency(attr->sched_latency_nice);
                          ^~~~~~~~~~~~~~
   kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   kernel/sched/sched.h:2483:31: note: in definition of macro 'bgtask_latency'
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                  ^~~
   kernel/sched/core.c:5018:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) != attr_leniency)
         ^~~~~~~~~~

vim +/unlikely +3272 kernel/sched/core.c

3190
3191 /**
3192 * finish_task_switch - clean up after a task-switch
3193 * @prev: the thread we just switched away from.
3194 *
3195 * finish_task_switch must be called after the context switch, paired
3196 * with a prepare_task_switch call before the context switch.
3197 * finish_task_switch will reconcile locking set up by prepare_task_switch,
3198 * and do any other architecture-specific cleanup actions.
3199 *
3200 * Note that we may have delayed dropping an mm in context_switch(). If
3201 * so, we finish that here outside of the runqueue lock. (Doing it
3202 * with the lock held can cause deadlocks; see schedule() for
3203 * details.)
3204 *
3205 * The context switch have flipped the stack from under us and restored the
3206 * local variables which were saved when this task called schedule() in the
3207 * past. prev == current is still correct but we need to recalculate this_rq
3208 * because prev may have moved to another CPU.
3209 */
3210 static struct rq *finish_task_switch(struct task_struct *prev)
3211 __releases(rq->lock)
3212 {
3213 struct rq *rq = this_rq();
3214 struct mm_struct *mm = rq->prev_mm;
3215 long prev_state;
3216
3217 /*
3218 * The previous task will have left us with a preempt_count of 2
3219 * because it left us after:
3220 *
3221 * schedule()
3222 * preempt_disable(); // 1
3223 * __schedule()
3224 * raw_spin_lock_irq(&rq->lock) // 2
3225 *
3226 * Also, see FORK_PREEMPT_COUNT.
3227 */
3228 if (WARN_ONCE(preempt_count() != 2*PREEMPT_DISABLE_OFFSET,
3229 "corrupted preempt_count: %s/%d/0x%x\n",
3230 current->comm, current->pid, preempt_count()))
3231 preempt_count_set(FORK_PREEMPT_COUNT);
3232
3233 rq->prev_mm = NULL;
3234
3235 /*
3236 * A task struct has one reference for the use as "current".
3237 * If a task dies, then it sets TASK_DEAD in tsk->state and calls
3238 * schedule one last time. The schedule call will never return, and
3239 * the scheduled task must drop that reference.
3240 *
3241 * We must observe prev->state before clearing prev->on_cpu (in
3242 * finish_task), otherwise a concurrent wakeup can get prev
3243 * running on another CPU and we could rave with its RUNNING -> DEAD
3244 * transition, resulting in a double drop.
3245 */
3246 prev_state = prev->state;
3247 vtime_task_switch(prev);
3248 perf_event_task_sched_in(prev, current);
3249 finish_task(prev);
3250 finish_lock_switch(rq);
3251 finish_arch_post_lock_switch();
3252 kcov_finish_switch(current);
3253
3254 fire_sched_in_preempt_notifiers(current);
3255 /*
3256 * When switching through a kernel thread, the loop in
3257 * membarrier_{private,global}_expedited() may have observed that
3258 * kernel thread and not issued an IPI. It is therefore possible to
3259 * schedule between user->kernel->user threads without passing though
3260 * switch_mm(). Membarrier requires a barrier after storing to
3261 * rq->curr, before returning to userspace, so provide them here:
3262 *
3263 * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly
3264 * provided by mmdrop(),
3265 * - a sync_core for SYNC_CORE.
3266 */
3267 if (mm) {
3268 membarrier_mm_sync_core_before_usermode(mm);
3269 mmdrop(mm);
3270 }
3271 if (unlikely(prev_state == TASK_DEAD)) {

> 3272			if (unlikely(is_bg_task(prev)))

3273 turbo_sched_put();
3274
3275 if (prev->sched_class->task_dead)
3276 prev->sched_class->task_dead(prev);
3277
3278 /*
3279 * Remove function-return probe instances associated with this
3280 * task and put them back on the free list.
3281 */
3282 kprobe_flush_task(prev);
3283
3284 /* Task is done with its stack. */
3285 put_task_stack(prev);
3286
3287 put_task_struct_rcu_user(prev);
3288 }
3289
3290 tick_nohz_task_switch();
3291 return rq;
3292 }
3293

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#7 lore raw

Re: [RFC v6 2/5] sched/core: Update turbo_sched count only when required

In reply to Parth Shah
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Parth,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on powerpc/next tip/auto-latest linux/master linus/master v5.5-rc7 next-20200121] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Parth-Shah/TurboSched-A-scheduler-for-sustaining-Turbo-Frequencies-for-longer-durations/20200124-041428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git afa70d941f663c69c9a64ec1021bbcfa82f0e54a
config: arm64-randconfig-a001-20200124 (attached as .config)
compiler: aarch64-linux-gcc (GCC) 7.5.0

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=arm64

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

kernel/sched/core.c:98:6: error: redefinition of 'turbo_sched_get'

    void turbo_sched_get(void) { return ; }
         ^~~~~~~~~~~~~~~
   kernel/sched/core.c:97:6: note: previous definition of 'turbo_sched_get' was here
    void turbo_sched_get(void) { return ; }
         ^~~~~~~~~~~~~~~
   In file included from arch/arm64/include/asm/current.h:5:0,
                    from include/linux/sched.h:12,
                    from kernel/sched/sched.h:5,
                    from kernel/sched/core.c:9:
   kernel/sched/core.c: In function 'finish_task_switch':
>> kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:78:42: note: in definition of macro 'unlikely'
    # define unlikely(x) __builtin_expect(!!(x), 0)
                                             ^
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
>> kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_CFTYPE_NAME'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:78:42: note: in definition of macro 'unlikely'
    # define unlikely(x) __builtin_expect(!!(x), 0)
                                             ^
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: note: each undeclared identifier is reported only once for each function it appears in
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:78:42: note: in definition of macro 'unlikely'
    # define unlikely(x) __builtin_expect(!!(x), 0)
                                             ^
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/core.c:3272:16: note: in expansion of macro 'is_bg_task'
      if (unlikely(is_bg_task(prev)))
                   ^~~~~~~~~~
   In file included from kernel/sched/core.c:9:0:
   kernel/sched/core.c: In function '__sched_setscheduler':
   kernel/sched/core.c:4801:44: error: 'const struct sched_attr' has no member named 'sched_latency_nice'; did you mean 'sched_nice'?
     bool attr_leniency = bgtask_latency(attr->sched_latency_nice);
                                               ^
   kernel/sched/sched.h:2483:31: note: in definition of macro 'bgtask_latency'
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                  ^~~
>> kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_CFTYPE_NAME'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   kernel/sched/core.c:4801:23: note: in expansion of macro 'bgtask_latency'
     bool attr_leniency = bgtask_latency(attr->sched_latency_nice);
                          ^~~~~~~~~~~~~~
>> kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   kernel/sched/sched.h:2483:31: note: in definition of macro 'bgtask_latency'
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                  ^~~
   kernel/sched/core.c:5018:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) != attr_leniency)
         ^~~~~~~~~~

vim +2484 kernel/sched/sched.h

2482

> 2483	#define bgtask_latency(lat)	((lat) == MAX_LATENCY_NICE)
> 2484	#define is_bg_task(p)		(bgtask_latency((p)->latency_nice))

2485

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

author
#8 lore raw

RFC v6 3/5 sched/fair: Tune task wake-up logic to pack small background tasks on fewer cores

In reply to Parth Shah

The algorithm finds the first non idle core in the system and tries to place a task in the idle CPU of the chosen core. To maintain cache hotness, work of finding non idle core starts from the prev_cpu, which also reduces task ping-pong behaviour inside of the core.

Define a new method to select_non_idle_core which keep tracks of the idle and non-idle CPUs in the core and based on the heuristics determines if the core is sufficiently busy to place the waking up background task. The heuristic further defines the non-idle CPU into either busy (>12.5% util) CPU and overutilized (>80% util) CPU.
- The core containing more idle CPUs and no busy CPUs is not selected for packing
- The core if contains more than 1 overutilized CPUs are exempted from task packing
- Pack if there is atleast one busy CPU and overutilized CPUs count is <2

Value of 12.5% utilization for busy CPU gives sufficient heuristics for CPU doing enough work and not become idle in nearby time frame.

---
 kernel/sched/core.c |  3 ++
 kernel/sched/fair.c | 87 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 89 insertions(+), 1 deletion(-)
kernel/sched/core.c +3 -0
index 629c2589d727..a34a5589ae16 100644
@@ -6617,6 +6617,7 @@ static struct kmem_cache *task_group_cache __read_mostly;
 
 DECLARE_PER_CPU(cpumask_var_t, load_balance_mask);
 DECLARE_PER_CPU(cpumask_var_t, select_idle_mask);
+DECLARE_PER_CPU(cpumask_var_t, turbo_sched_mask);
 
 void __init sched_init(void)
 {
@@ -6657,6 +6658,8 @@ void __init sched_init(void)
 			cpumask_size(), GFP_KERNEL, cpu_to_node(i));
 		per_cpu(select_idle_mask, i) = (cpumask_var_t)kzalloc_node(
 			cpumask_size(), GFP_KERNEL, cpu_to_node(i));
+		per_cpu(turbo_sched_mask, i) = (cpumask_var_t)kzalloc_node(
+			cpumask_size(), GFP_KERNEL, cpu_to_node(i));
 	}
 #endif /* CONFIG_CPUMASK_OFFSTACK */
 
kernel/sched/fair.c +86 -1

111 lines not on this page · raw message

Signature · 1 line
2.17.2
#9 lore raw

Re: [RFC v6 3/5] sched/fair: Tune task wake-up logic to pack small background tasks on fewer cores

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Parth,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on powerpc/next tip/auto-latest linux/master linus/master v5.5-rc7 next-20200123] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Parth-Shah/TurboSched-A-scheduler-for-sustaining-Turbo-Frequencies-for-longer-durations/20200124-041428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git afa70d941f663c69c9a64ec1021bbcfa82f0e54a
config: ia64-randconfig-a001-20200124 (attached as .config)
compiler: ia64-linux-gcc (GCC) 7.5.0

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=ia64

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

In file included from kernel/sched/fair.c:23:0:
kernel/sched/fair.c: In function 'is_small_bg_task':
kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?

    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   kernel/sched/sched.h:2483:31: note: in definition of macro 'bgtask_latency'
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                  ^~~
>> kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_CFTYPE_NAME'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
>> kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: note: each undeclared identifier is reported only once for each function it appears in
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
>> kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~

vim +/is_bg_task +5910 kernel/sched/fair.c

5903
5904 /*
5905 * Classify small background tasks with higher latency_nice value for task
5906 * packing.
5907 */
5908 static inline bool is_small_bg_task(struct task_struct *p)
5909 {

> 5910		if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))

5911 return true;
5912
5913 return false;
5914 }
5915

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#10 lore raw

Re: [RFC v6 3/5] sched/fair: Tune task wake-up logic to pack small background tasks on fewer cores

In reply to Parth Shah
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Parth,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on powerpc/next tip/auto-latest linux/master linus/master v5.5-rc7 next-20200122] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Parth-Shah/TurboSched-A-scheduler-for-sustaining-Turbo-Frequencies-for-longer-durations/20200124-041428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git afa70d941f663c69c9a64ec1021bbcfa82f0e54a
config: sparc64-randconfig-a001-20200124 (attached as .config)
compiler: sparc64-linux-gcc (GCC) 7.5.0

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=sparc64

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

In file included from arch/sparc/include/asm/bug.h:6:0,
from include/linux/bug.h:5,
from include/linux/thread_info.h:12,
from arch/sparc/include/asm/current.h:15,
from include/linux/sched.h:12,
from kernel/sched/sched.h:5,
from kernel/sched/fair.c:23:
kernel/sched/fair.c: In function 'is_small_bg_task':
kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?

    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:58:52: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                       ^~~~
>> kernel/sched/fair.c:5910:2: note: in expansion of macro 'if'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
     ^~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: error: 'MAX_LATENCY_NICE' undeclared (first use in this function); did you mean 'MAX_CFTYPE_NAME'?
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:58:52: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                       ^~~~
>> kernel/sched/fair.c:5910:2: note: in expansion of macro 'if'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
     ^~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2483:39: note: each undeclared identifier is reported only once for each function it appears in
    #define bgtask_latency(lat) ((lat) == MAX_LATENCY_NICE)
                                          ^
   include/linux/compiler.h:58:52: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                       ^~~~
>> kernel/sched/fair.c:5910:2: note: in expansion of macro 'if'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
     ^~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:58:61: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                                ^~~~
>> kernel/sched/fair.c:5910:2: note: in expansion of macro 'if'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
     ^~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~
   kernel/sched/sched.h:2484:45: error: 'struct task_struct' has no member named 'latency_nice'; did you mean 'latency_record'?
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                                                ^
   include/linux/compiler.h:69:3: note: in definition of macro '__trace_if_value'
     (cond) ?     \
      ^~~~
   include/linux/compiler.h:56:28: note: in expansion of macro '__trace_if_var'
    #define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
                               ^~~~~~~~~~~~~~
>> kernel/sched/fair.c:5910:2: note: in expansion of macro 'if'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
     ^~
   kernel/sched/sched.h:2484:25: note: in expansion of macro 'bgtask_latency'
    #define is_bg_task(p)  (bgtask_latency((p)->latency_nice))
                            ^~~~~~~~~~~~~~
   kernel/sched/fair.c:5910:6: note: in expansion of macro 'is_bg_task'
     if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))
         ^~~~~~~~~~

vim +/if +5910 kernel/sched/fair.c

5903
5904 /*
5905 * Classify small background tasks with higher latency_nice value for task
5906 * packing.
5907 */
5908 static inline bool is_small_bg_task(struct task_struct *p)
5909 {

> 5910		if (is_bg_task(p) && (task_util(p) > (SCHED_CAPACITY_SCALE >> 3)))

5911 return true;
5912
5913 return false;
5914 }
5915

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#11 lore raw

[RFC PATCH] sched/fair: __pcpu_scope_turbo_sched_mask can be static

In reply to Parth Shah
Fixes: 29ef03d9d30c ("sched/fair: Tune task wake-up logic to pack small background tasks on fewer cores")
Fixes 29ef03d9d30c ("sched/fair: Tune task wake-up logic to pack small background tasks on fewer cores")
---
 fair.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index 230098e71a9ec..06b73b4fad098 100644
@@ -5400,7 +5400,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 DEFINE_PER_CPU(cpumask_var_t, load_balance_mask);
 DEFINE_PER_CPU(cpumask_var_t, select_idle_mask);
 /* A cpumask to find active cores in the system. */
-DEFINE_PER_CPU(cpumask_var_t, turbo_sched_mask);
+static DEFINE_PER_CPU(cpumask_var_t, turbo_sched_mask);
 
 #ifdef CONFIG_NO_HZ_COMMON
 
author
#12 lore raw

RFC v6 4/5 sched/fair: Provide arch hook to find domain for non idle core search scan

In reply to Parth Shah

Specify the method which returns cpumask within which to limit the search for a non idle core. By default, limit the search in LLC domain which usually includes few/all the cores in the processor chip.

The select_non_idle_core searches for the non idle cores in the LLC domain. But in the systems with multiple NUMA domains, the Turbo frequency can be sustained within the NUMA domain without being affected from other NUMA. For such case, arch_turbo_domain can be tuned to change domain for non idle core search.

---
 kernel/sched/fair.c | 8 ++++++++
 1 file changed, 8 insertions(+)
kernel/sched/fair.c +8 -0
index 8643e6309451..af19e1f9d56d 100644
@@ -5890,6 +5890,13 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 /* Define non-idle CPU as the one with the utilization >= 12.5% */
 #define merely_used_cpu(util) ((cpu_util(util)) > (100 >> 3))
 
+#ifndef arch_turbo_domain
+static __always_inline struct cpumask *arch_turbo_domain(int cpu)
+{
+	return sched_domain_span(rcu_dereference(per_cpu(sd_llc, cpu)));
+}
+#endif
+
 /*
  * Classify small background tasks with higher latency_nice value for task
  * packing.
@@ -5916,6 +5923,7 @@ static int select_non_idle_core(struct task_struct *p, int prev_cpu)
 	int iter_cpu, sibling;
 
 	cpumask_and(cpus, cpu_online_mask, p->cpus_ptr);
+	cpumask_and(cpus, cpus, arch_turbo_domain(prev_cpu));
 
 	for_each_cpu_wrap(iter_cpu, cpus, prev_cpu) {
 		int idle_cpu_count = 0, non_idle_cpu_count = 0;
Signature · 1 line
2.17.2
author
#13 lore raw

RFC v6 5/5 powerpc: Set turbo domain to NUMA node for task packing

In reply to Parth Shah

Provide an powerpc architecture specific implementation for defining the turbo domain to make searching of the core to be bound within the NUMA.

The POWER9 systems have a pair of cores in the LLC domain. Hence to make TurboSched more effective, increase the domain space for task packing to search within NUMA domain.

---
 arch/powerpc/include/asm/topology.h | 3 +++
 arch/powerpc/kernel/smp.c           | 7 +++++++
 2 files changed, 10 insertions(+)
arch/powerpc/include/asm/topology.h +3 -0
index 2f7e1ea5089e..83adfb99f8ba 100644
@@ -138,6 +138,9 @@ static inline void shared_proc_topology_init(void) {}
 #define topology_sibling_cpumask(cpu)	(per_cpu(cpu_sibling_map, cpu))
 #define topology_core_cpumask(cpu)	(per_cpu(cpu_core_map, cpu))
 #define topology_core_id(cpu)		(cpu_to_core_id(cpu))
+#define arch_turbo_domain		powerpc_turbo_domain
+
+struct cpumask *powerpc_turbo_domain(int cpu);
 
 int dlpar_cpu_readd(int cpu);
 #endif
arch/powerpc/kernel/smp.c +7 -0
index ea6adbf6a221..0fc4443a3f27 100644
@@ -1169,6 +1169,13 @@ static void remove_cpu_from_masks(int cpu)
 }
 #endif
 
+#ifdef CONFIG_SCHED_SMT
+inline struct cpumask *powerpc_turbo_domain(int cpu)
+{
+	return cpumask_of_node(cpu_to_node(cpu));
+}
+#endif
+
 static inline void add_cpu_to_smallcore_masks(int cpu)
 {
 	struct cpumask *this_l1_cache_map = per_cpu(cpu_l1_cache_map, cpu);
Signature · 1 line
2.17.2