sched/fair: Complete EEVDF
Peter Zijlstra
maintainercover sched/fair: Complete EEVDF
Hi all,
I'm slowly crawling back out of the hole and trying to get back to work. Availability is still low on my end, but I'll try and respond to some email.
Anyway, in order to get my feet wet again with sitting behind a computer, find here a few patches that should functionally complete the EEVDF journey.
This very much includes the new interface that exposes the extra parameter that EEVDF has. I've chosen to use sched_attr::sched_runtime for this over a nice-like value because some workloads actually know their slice length (can be dynamically measured in the same way as for deadline using CLOCK_THREAD_CPUTIME_ID) and using the real request size is much more effective than some relative measure.
[[ using too short a request size will increase job preemption overhead,
using too long a request size will decrease timeliness ]]
The whole delayed-dequeue thing is I think a fundamental thing that was missing from the EEVDF paper. Without something like this EEVDF will simply not work right. IIRC this was mentioned to me many years ago when people worked on BFQ iosched and ran into this same issue. Time had erased the critical aspect of this note and I had to re-discover it again.
Also, I think Ben expressed concern that preserving lag over long periods doesn't make sense a while back.
The implementation presented here is one that should work with our cgroup mess and keeps most of the ugly inside fair.c unlike previous versions which puked all over the core scheduler code.
Critically cfs-cgroup throttling is not tested, and cgroups are only tested in so far that a systemd infected machine now boots (took a bit).
Other than that, it works well enough to build the next kernel and it passes the few trivial latency-slice tests I ran.
Anyway, please have a poke and let me know...
Peter Zijlstra
maintainerRFC PATCH 1/10 sched/eevdf: Add feature comments
--- kernel/sched/features.h | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/sched/features.h +7 -0
@@ -5,7 +5,14 @@* sleep+wake cycles. EEVDF placement strategy #1, #2 if disabled. */ SCHED_FEAT (PLACE_LAG , true) +/* +* Give new tasks half a slice to ease into the competition. + */SCHED_FEAT (PLACE_DEADLINE_INITIAL , true) +/* +* Inhibit (wakeup) preemption until the current task has either matched the +* 0-lag point or until is has exhausted it's slice. + */SCHED_FEAT (RUN_TO_PARITY , true)/*
Peter Zijlstra
maintainerRFC PATCH 2/10 sched/eevdf: Remove min_vruntime_copy
In reply to Peter ZijlstraSince commit e8f331bcc270 ("sched/smp: Use lag to simplify cross-runqueue placement") the min_vruntime_copy is no longer used.
--- kernel/sched/fair.c | 5 ++--- kernel/sched/sched.h | 4 ---- 2 files changed, 2 insertions(+), 7 deletions(-)
kernel/sched/fair.c +2 -3
@@ -780,8 +780,7 @@ static void update_min_vruntime(struct c }/* ensure we never gain time by being placed backwards. */ -u64_u32_store (cfs_rq->min_vruntime, -__update_min_vruntime (cfs_rq, vruntime)); + cfs_rq->min_vruntime =__update_min_vruntime (cfs_rq, vruntime); }static inline bool __entity_less (struct rb_node *a,const struct rb_node *b)
@@ -12876,7 +12875,7 @@ static void set_next_task_fair(struct rqvoid init_cfs_rq (struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =RB_ROOT_CACHED ; -u64_u32_store (cfs_rq->min_vruntime, (u64)(-(1LL <<20 ))); + cfs_rq->min_vruntime = (u64)(-(1LL <<20 ));#ifdef CONFIG_SMP raw_spin_lock_init (&cfs_rq->removed.lock);#endif
kernel/sched/sched.h +0 -4
@@ -599,10 +599,6 @@ struct cfs_rq {u64 min_vruntime_fi;#endif -#ifndef CONFIG_64BIT -u64 min_vruntime_copy; -#endif -struct rb_root_cached tasks_timeline;/*
Peter Zijlstra
maintainerRFC PATCH 3/10 sched/fair: Cleanup pick_task_fair() vs throttle
In reply to Peter ZijlstraPer 54d27365cae8 ("sched/fair: Prevent throttling in early pick_next_task_fair()") the reason check_cfs_rq_runtime() is under the 'if (curr)' check is to ensure the (downward) traversal does not result in an empty cfs_rq.
But then the pick_task_fair() 'copy' of all this made it restart the traversal anyway, so that seems to solve the issue too.
--- kernel/sched/fair.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/fair.c +3 -3
@@ -8435,11 +8435,11 @@ static struct task_struct *pick_task_faiupdate_curr (cfs_rq); else curr =NULL ; - -if (unlikely (check_cfs_rq_runtime (cfs_rq))) - goto again; } +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + se =pick_next_entity (cfs_rq); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Re: [RFC][PATCH 03/10] sched/fair: Cleanup pick_task_fair() vs throttle
Quoted 7 lines · Peter Zijlstra
Per
54d27365cae8("sched/fair: Prevent throttling in earlypick_next_task_fair()") the reasoncheck_cfs_rq_runtime()is under the 'if (curr)' check is to ensure the (downward) traversal does not result in an emptycfs_rq.But then the
pick_task_fair()'copy' of all this made it restart the traversal anyway, so that seems to solve the issue too.
Yeah, putting the check_cfs_rq_runtime inside of that condition was specific to the exact pnt_fair code, and the specific places that nr_running was and was not checked. pick_task_fair doesn't care about any of that, and if instead put_prev manages to throttle the picked task, we can still successfully switch to it for a moment.
Quoted 23 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/sched/fair.c +3
-3@@ -8435,11 +8435,11 @@ static struct task_struct *pick_task_faiupdate_curr (cfs_rq); else curr =NULL ; - -if (unlikely (check_cfs_rq_runtime (cfs_rq))) - goto again; } +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + se =pick_next_entity (cfs_rq); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Peter Zijlstra
maintainerRFC PATCH 4/10 sched/fair: Cleanup pick_task_fair()s curr
In reply to Peter ZijlstraWith 4c456c9ad334 ("sched/fair: Remove unused 'curr' argument from pick_next_entity()") curr is no longer being used, so no point in clearing it.
--- kernel/sched/fair.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
kernel/sched/fair.c +2 -8
@@ -8427,15 +8427,9 @@ static struct task_struct *pick_task_faireturn NULL ; do { - struct sched_entity *curr = cfs_rq->curr; - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ - if (curr) { - if (curr->on_rq) - update_curr(cfs_rq); - else - curr = NULL; - } + if (cfs_rq->curr && cfs_rq->curr->on_rq) + update_curr(cfs_rq); if (unlikely(check_cfs_rq_runtime(cfs_rq))) goto again;
Peter Zijlstra
maintainerRFC PATCH 5/10 sched/fair: Unify pick_{,next_}_task_fair()
In reply to Peter ZijlstraImplement pick_next_task_fair() in terms of pick_task_fair() to de-duplicate the pick loop.
More importantly, this makes all the pick loops use the state-invariant form, which is useful to introduce further re-try conditions in later patches.
--- kernel/sched/fair.c | 60 ++++++---------------------------------------------- 1 file changed, 8 insertions(+), 52 deletions(-)
kernel/sched/fair.c +8 -52
@@ -8415,7 +8415,6 @@ static void check_preempt_wakeup_fair(stresched_curr (rq); } -#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) {struct sched_entity *se;
@@ -8427,7 +8426,7 @@ static struct task_struct *pick_task_faireturn NULL ; do { - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ + /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) update_curr(cfs_rq);
@@ -8440,19 +8439,20 @@ static struct task_struct *pick_task_faireturn task_of (se); } -#endif struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se;struct task_struct *p; +struct cfs_rq *cfs_rq;int new_tasks; again: -if (!sched_fair_runnable (rq)) + p =pick_task_fair (rq); +if (!p) goto idle; + se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED if (!prev || prev->sched_class != &fair_sched_class)
@@ -8464,46 +8464,7 @@ pick_next_task_fair(struct rq *rq, struc* * Therefore attempt to avoid putting and setting the entire cgroup * hierarchy, only change the part that actually changes. -*/ - -do { -struct sched_entity *curr = cfs_rq->curr; - -/* -* Since we got here without doing put_prev_entity() we also -* have to consider cfs_rq->curr. If it is still a runnable -* entity, update_curr() will update its vruntime, otherwise -* forget we've ever seen it. -*/ -if (curr) { -if (curr->on_rq) -update_curr (cfs_rq); -else - curr =NULL ; - -/* -* This call to check_cfs_rq_runtime() will do the -* throttle and dequeue its entity in the parent(s). -* Therefore the nr_running test will indeed -* be correct. -*/ -if (unlikely (check_cfs_rq_runtime (cfs_rq))) { - cfs_rq = &rq->cfs; - -if (!cfs_rq->nr_running) - goto idle; - - goto simple; - } - } - - se =pick_next_entity (cfs_rq); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); - -/* +* * Since we haven't yet done put_prev_entity and if the selected task * is a different task than we started out with, try and touch the * least amount of cfs_rqs.
@@ -8535,13 +8496,8 @@ pick_next_task_fair(struct rq *rq, strucif (prev)put_prev_task (rq, prev); -do { - se =pick_next_entity (cfs_rq); -set_next_entity (cfs_rq, se); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); +for_each_sched_entity (se) +set_next_entity (cfs_rq_of (se), se); done:__maybe_unused ;#ifdef CONFIG_SMP
Re: [RFC][PATCH 05/10] sched/fair: Unify pick_{,next_}_task_fair()
Greetings (good to see you back).
Quoted 16 lines · Peter Zijlstra
(quoted diff) +0
-0@@ -8440,19 +8439,20 @@ static struct task_struct *pick_task_fai
returntask_of(se);
}
-#endif
structtask_struct*
pick_next_task_fair(struct rq *rq, structtask_struct*prev, structrq_flags*rf) {
- structcfs_rq*cfs_rq=&rq->cfs;
structsched_entity*se;
structtask_struct*p;
+ structcfs_rq*cfs_rq;
intnew_tasks;
again:
Quoted 8 lines
- if (!sched_fair_runnable(rq))
+ p =pick_task_fair(rq);
+ if (!p)
goto idle;
+ se =&p->se;
#ifdefCONFIG_FAIR_GROUP_SCHED
if (!prev ||prev->sched_class!=&fair_sched_class)
Those who dodge GROUP_SCHED overhead receive a shiny new unused variable warning.
-Mike
Peter Zijlstra
maintainerRFC PATCH 6/10 sched: Allow sched_class::dequeue_task() to fail
In reply to Peter ZijlstraChange the function signature of sched_class::dequeue_task() to return a boolean, allowing future patches to 'fail' dequeue.
--- kernel/sched/core.c | 7 +++++-- kernel/sched/deadline.c | 4 +++- kernel/sched/fair.c | 4 +++- kernel/sched/idle.c | 3 ++- kernel/sched/rt.c | 4 +++- kernel/sched/sched.h | 2 +- kernel/sched/stop_task.c | 3 ++- 7 files changed, 19 insertions(+), 8 deletions(-)
kernel/sched/core.c +5 -2
@@ -2119,7 +2119,10 @@ static inline void enqueue_task(struct rsched_core_enqueue (rq, p); } -static inline void dequeue_task (struct rq *rq,struct task_struct *p,int flags) +/* +* Must only return false when DEQUEUE_SLEEP. +*/ +static inline bool dequeue_task (struct rq *rq,struct task_struct *p,int flags) {if (sched_core_enabled (rq))sched_core_dequeue (rq, p, flags);
@@ -2133,7 +2136,7 @@ static inline void dequeue_task(struct r }uclamp_rq_dec (rq, p); - p->sched_class->dequeue_task (rq, p, flags); +return p->sched_class->dequeue_task (rq, p, flags); }void activate_task (struct rq *rq,struct task_struct *p,int flags)
kernel/sched/deadline.c +3 -1
@@ -1842,7 +1842,7 @@ static void enqueue_task_dl(struct rq *renqueue_pushable_dl_task (rq, p); } -static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {update_curr_dl (rq);
@@ -1852,6 +1852,8 @@ static void dequeue_task_dl(struct rq *rdequeue_dl_entity (&p->dl, flags);if (!p->dl.dl_throttled && !dl_server (&p->dl))dequeue_pushable_dl_task (rq, p); + +return true; }/*
kernel/sched/fair.c +3 -1
@@ -6828,7 +6828,7 @@ static void set_next_buddy(struct sched_* decreased. We remove the task from the rbtree and * update the fair scheduling stats: */ -static void dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) {struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;
@@ -6896,6 +6896,8 @@ static void dequeue_task_fair(struct rq dequeue_throttle:util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); + +return true; }#ifdef CONFIG_SMP
kernel/sched/idle.c +2 -1
@@ -486,13 +486,14 @@ struct task_struct *pick_next_task_idle(* It is not legal to sleep in the idle task - print a warning * message if some code attempts to do it: */ -static void +static bool dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) {raw_spin_rq_unlock_irq (rq);printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack ();raw_spin_rq_lock_irq (rq); +return true; }/*
kernel/sched/rt.c +3 -1
@@ -1493,7 +1493,7 @@ enqueue_task_rt(struct rq *rq, struct taenqueue_pushable_task (rq, p); } -static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt;
@@ -1501,6 +1501,8 @@ static void dequeue_task_rt(struct rq *rdequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +return true; }/*
kernel/sched/sched.h +1 -1
@@ -2278,7 +2278,7 @@ struct sched_class {#endif void (*enqueue_task) (struct rq *rq,struct task_struct *p,int flags); -void (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags); +bool (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags);void (*yield_task) (struct rq *rq);bool (*yield_to_task)(struct rq *rq,struct task_struct *p);
kernel/sched/stop_task.c +2 -1
@@ -57,10 +57,11 @@ enqueue_task_stop(struct rq *rq, structadd_nr_running (rq,1 ); } -static void +static bool dequeue_task_stop (struct rq *rq,struct task_struct *p,int flags) {sub_nr_running (rq,1 ); +return true; }static void yield_task_stop (struct rq *rq)
Peter Zijlstra
maintainerRFC PATCH 7/10 sched/fair: Re-organize dequeue_task_fair()
In reply to Peter ZijlstraWorking towards delaying dequeue, notably also inside the hierachy, rework dequeue_task_fair() such that it can 'resume' an interrupted hierarchy walk.
--- kernel/sched/fair.c | 82 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 55 insertions(+), 27 deletions(-)
kernel/sched/fair.c +55 -27
@@ -6824,33 +6824,45 @@ enqueue_task_fair(struct rq *rq, structstatic void set_next_buddy (struct sched_entity *se);/* -* The dequeue_task method is called before nr_running is -* decreased. We remove the task from the rbtree and -* update the fair scheduling stats: +* Basically dequeue_task_fair(), except it can deal with dequeue_entity() +* failing half-way through and resume the dequeue later. +* +* Returns: +* -1 - dequeue delayed +* 0 - dequeue throttled +* 1 - dequeue complete */ -static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq); +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ; +struct cfs_rq *cfs_rq; +int idle_h_nr_running; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + idle_h_nr_running =task_has_idle_policy (p); + }else { + idle_h_nr_running =cfs_rq_is_idle (group_cfs_rq (se)); + }for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);dequeue_entity (cfs_rq, se, flags); - cfs_rq->h_nr_running--; - cfs_rq->idle_h_nr_running -= idle_h_nr_running; +/* h_nr_running is the hierachical count of tasks */ +if (p) { + cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; -if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; + }/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; +return 0 ;/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {
@@ -6870,33 +6882,49 @@ static bool dequeue_task_fair(struct rqfor_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); +// XXX avoid these load updates for delayed dequeues ? update_load_avg (cfs_rq, se,UPDATE_TG );se_update_runnable (se);update_cfs_group (se); - cfs_rq->h_nr_running--; - cfs_rq->idle_h_nr_running -= idle_h_nr_running; +if (p) { + cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; -if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; +if (cfs_rq_is_idle (cfs_rq)) + idle_h_nr_running =1 ; + }/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; +return 0 ; + } + +if (p) { +sub_nr_running (rq,1 ); +/* balance early to pull high priority tasks */ +if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) + rq->next_balance = jiffies; } -/* At this point se is NULL and we are at root level*/ -sub_nr_running (rq,1 ); +return 1 ; +} -/* balance early to pull high priority tasks */ -if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) - rq->next_balance = jiffies; +/* +* The dequeue_task method is called before nr_running is +* decreased. We remove the task from the rbtree and +* update the fair scheduling stats: +*/ +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +{ +util_est_dequeue (&rq->cfs, p); -dequeue_throttle: -util_est_update (&rq->cfs, p, task_sleep); -hrtick_update (rq); +if (dequeue_entities (rq, &p->se, flags) <0 ) +return false; +util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP ); +hrtick_update (rq);return true; }
Peter Zijlstra
maintainerRFC PATCH 8/10 sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraExtend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.
OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
XXX test the cfs-throttle stuff
--- include/linux/sched.h | 1 kernel/sched/core.c | 22 +++++-- kernel/sched/fair.c | 148 +++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/features.h | 12 +++ kernel/sched/sched.h | 2 5 files changed, 167 insertions(+), 18 deletions(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {struct list_head group_node;unsigned int on_rq; +unsigned int sched_delayed;u64 exec_start;u64 sum_exec_runtime;
kernel/sched/core.c +18 -4
@@ -2154,10 +2154,14 @@ void activate_task(struct rq *rq, structvoid deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -WRITE_ONCE (p->on_rq, (flags &DEQUEUE_SLEEP ) ?0 :TASK_ON_RQ_MIGRATING ); -ASSERT_EXCLUSIVE_WRITER (p->on_rq); +bool sleep = flags &DEQUEUE_SLEEP ; -dequeue_task (rq, p, flags); +if (dequeue_task (rq, p, flags)) { +WRITE_ONCE (p->on_rq, sleep ?0 :TASK_ON_RQ_MIGRATING ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); + }else { +SCHED_WARN_ON (!sleep);/* only sleep can fail */ + } }static inline int __normal_prio (int policy,int rt_prio,int nice)
@@ -3858,12 +3862,17 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) { +update_rq_clock (rq); +if (p->se.sched_delayed) { +/* mustn't run a delayed task */ +SCHED_WARN_ON (task_on_cpu (rq, p)); +enqueue_task (rq, p,ENQUEUE_DELAYED ); + }if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -update_rq_clock (rq);wakeup_preempt (rq, p, wake_flags); }ttwu_do_wakeup (p);
@@ -4243,11 +4252,16 @@ int try_to_wake_up(struct task_struct *p* case the whole 'p->on_rq && ttwu_runnable()' case below * without taking any locks. * +* Specifically, given current runs ttwu() we must be before +* schedule()'s deactivate_task(), as such this must not +* observe sched_delayed. +* * In particular: * - we rely on Program-Order guarantees for all the ordering, * - we're serialized against set_special_state() by virtue of * it disabling IRQs (this allows not taking ->pi_lock). */ +SCHED_WARN_ON (p->se.sched_delayed);if (!ttwu_state_match (p, state, &success)) goto out;
kernel/sched/fair.c +134 -14
280 lines not on this page · raw message
kernel/sched/features.h +12 -0
@@ -29,6 +29,18 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +* +* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. +* +* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. + */ +SCHED_FEAT(DELAY_DEQUEUE, true) +SCHED_FEAT (DELAY_ZERO , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
kernel/sched/sched.h +2 -0
@@ -2245,6 +2245,7 @@ extern const u32 sched_prio_to_wmult[40#define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ #define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ +#define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */ #define ENQUEUE_WAKEUP 0x01#define ENQUEUE_RESTORE 0x02
@@ -2260,6 +2261,7 @@ extern const u32 sched_prio_to_wmult[40#endif #define ENQUEUE_INITIAL 0x80#define ENQUEUE_MIGRATING 0x100 +#define ENQUEUE_DELAYED 0x200#define RETRY_TASK ((void *)-1UL)
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 19 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Quoted 7 lines
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
XXX test the cfs-throttle stuff
Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
Tested schbench on xeon server, which has 240 CPUs/2 sockets. schbench -m 2 -r 100
the result seems ok to me.
baseline:NO_DELAY_DEQUEUENO_DELAY_ZERO
Wakeup Latencies percentiles (usec) runtime 100 (s) (1658446 total samples)
50.0th: 5 (361126 samples) 90.0th: 11 (654121 samples) * 99.0th: 25 (123032 samples) 99.9th: 673 (13845 samples) min=1, max=8337 Request Latencies percentiles (usec) runtime 100 (s) (1662381 total samples) 50.0th: 14992 (524771 samples) 90.0th: 15344 (657370 samples) * 99.0th: 15568 (129769 samples) 99.9th: 15888 (10017 samples) min=3529, max=43841 RPS percentiles (requests) runtime 100 (s) (101 total samples) 20.0th: 16544 (37 samples) * 50.0th: 16608 (30 samples) 90.0th: 16736 (31 samples) min=16403, max=17698
average rps: 16623.81
DELAY_DEQUEUEDELAY_ZERO
Wakeup Latencies percentiles (usec) runtime 100 (s) (1668161 total samples)
50.0th: 6 (394867 samples) 90.0th: 12 (653021 samples) * 99.0th: 31 (142636 samples) 99.9th: 755 (14547 samples) min=1, max=5226 Request Latencies percentiles (usec) runtime 100 (s) (1671859 total samples) 50.0th: 14384 (511809 samples) 90.0th: 14992 (653508 samples) * 99.0th: 15408 (149257 samples) 99.9th: 15984 (12090 samples) min=3546, max=38360 RPS percentiles (requests) runtime 100 (s) (101 total samples) 20.0th: 16672 (45 samples) * 50.0th: 16736 (52 samples) 90.0th: 16736 (0 samples) min=16629, max=16800
average rps: 16718.59
The 99th wakeup latency increases a little bit, and should be in the acceptible range(25 -> 31 us). Meanwhile the throughput increases accordingly. Here are the possible reason I can think of:
1. wakeup latency: The time to find an eligible entity in the tree during wakeup might take longer - if there are more delayed-dequeue tasks in the tree.
2. throughput: Inhibit task dequeue can decrease the ratio to touch the task group's load_avg: dequeue_entity()-> { update_load_avg(), update_cfs_group()), which reduces the cache contention among CPUs, and improves throughput.
kernel/sched/fair.c static void clear_buddies(struct cfs_rq · line 5379
Quoted 8 lines
kernel/sched/fair.c +8
-0+ }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); + +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) {
Regarding the elibigle check, it was found that there could be an overflow issue, and it brings false negative of entity_eligible(), which was described here:
https://lore.kernel.org/lkml/[email protected]/
and also reported on another machine
https://lore.kernel.org/lkml/[email protected]/
I don't have good idea to avoid that overflow properly, while I'm trying to reproduce it locally, do you have any guidance on how to address it?
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 2 lines · Chen Yu
The 99th wakeup latency increases a little bit, and should be in the acceptible
range(25 -> 31 us).
Ah, my test runs haven't been stable enough to observe that.
Quoted 6 lines
Meanwhile the throughput increases accordingly. Here are the possible reason I can think of:
1. wakeup latency: The time to find an eligible entity in the tree during wakeup might take longer - if there are more delayed-dequeue tasks in the tree.
Another possible cause might be that previously a schedule() would be 1 dequeue, 1 pick.
But now it can be much more variable, a pick can basically do N dequeues and N+1 picks.
So not only do we do more picks, but if you're focussed on worst case latency, it goes up, because we can do multiple dequeues for a single pick.
If we find this to really be a problem, I had some half baked ideas to fix it, but it added significant complexity, so keep it simple until need proves we need more etc.
Quoted 3 lines
2. throughput: Inhibit task dequeue can decrease the ratio to touch the task group's
load_avg: dequeue_entity()-> {update_load_avg(),update_cfs_group()), which reduces the cache contention among CPUs, and improves throughput.
Ah, yes, there's that.
Quoted 9 lines
kernel/sched/fair.c static void clear_buddies(struct cfs_rq · line 5379
Quoted 8 lines
kernel/sched/fair.c +8
-0+ }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); + +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) {
Quoted 7 lines
Regarding the elibigle check, it was found that there could be an overflow issue, and it brings false negative of
entity_eligible(), which was described here:
https://lore.kernel.org/lkml/[email protected]/
and also reported on another machine
https://lore.kernel.org/lkml/[email protected]/
I don't have good idea to avoid that overflow properly, while I'm trying to reproduce it locally, do you have any guidance on how to address it?
I have not yet seen those, let me go stare at them now. Thanks!
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 19 lines
Quoted 3 lines
The 99th wakeup latency increases a little bit, and should be in the acceptible
range(25 -> 31 us).Ah, my test runs haven't been stable enough to observe that.
Quoted 6 lines
Meanwhile the throughput increases accordingly. Here are the possible reason I can think of:
1. wakeup latency: The time to find an eligible entity in the tree during wakeup might take longer - if there are more delayed-dequeue tasks in the tree.
Another possible cause might be that previously a
schedule()would be 1 dequeue, 1 pick.But now it can be much more variable, a pick can basically do N dequeues and N+1 picks.
Quoted 7 lines
So not only do we do more picks, but if you're focussed on worst case latency, it goes up, because we can do multiple dequeues for a single pick.
If we find this to really be a problem, I had some half baked ideas to fix it, but it added significant complexity, so keep it simple until need proves we need more etc.
I have an alternative approach to delayed-dequeue inspired by the original CFS implementation.
The idea is to keep the task's vruntime when it goes to sleep.
When the task is woken up, see if the lag is positive at the woken time, if it is the case, clamp it to 0 by setting vruntime to avg_vruntime().
<Sleep>
In dequeue_entity(): Remove the task from runqueue, but keep the task's vruntime, and do not calculate vlag at this time.
<Wake Up on the same CPU>
In enqueue_entity():
1. Do not call place_entity().
2. If the task's vruntume is less than the cfs_rq's avg_vruntime(),
set the task's vruntime to avg_vruntime(), and update the task's deadline according to its timeslice.
3. Insert the task into the runqueue.
<Wake Up on different CPU>
In migrate_task_rq_fair():
1. Calculate the task's vlag as if it is on the original cfs_rq.
2. Set the task's vlag to 0 if it is positive.
In enqueue_entity(): Use place_entity() to calculate the task's new vruntime and deadline according to the vlag and the new runqueue before inserting it into the runqueue.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 6 lines · Yan-Jie Wang
I have an alternative approach to delayed-dequeue inspired by the original CFS implementation.
The idea is to keep the task's vruntime when it goes to sleep. When the task is woken up, see if the lag is positive at the woken time, if it is the case, clamp it to 0 by setting vruntime to
avg_vruntime().
Problem is that avg_vruntime() can go backwards, eg by dequeueing a negative lag task. This gets really hard to argue about real quick.
Keeping the task competing (delaying the dequeue) is by far the simplest solution -- conceptually.
The code just became a total mess because of cgroups :/
K Prateek Nayak
reviewerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHello Peter,
Quoted 15 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Quoted 8 lines
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
XXX test the cfs-throttle stuff
I ran into a few issues when testing the series on top of tip:sched/core at commit 4475cd8bfd9b ("sched/balancing: Simplify the sg_status bitmask and use separate ->overloaded and ->overutilized flags"). All of these splats surfaced when running unixbench with Delayed Dequeue (echoing NO_DELAY_DEQUEUE to /sys/kernel/debug/sched/features seems to make the system stable when running Unixbench spawn)
Unixbench (https://github.com/kdlucas/byte-unixbench.git) command:
./Run spawn -c 512
Splats appear soon into the run. Following are the splats and their corresponding code blocks from my 3rd Generation EPYC system (2 x 64C/128T):
1. NULL pointer dereferencing in can_migrate_task():
BUG: kernel NULL pointer dereference, address: 0000000000000040 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 154 PID: 1507736 Comm: spawn Not tainted 6.9.0-rc1-test+ #958 Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.7.3 03/30/2022 RIP: 0010:can_migrate_task+0x2b/0x6c0 Code: ... RSP: 0018:ffffb6bb9e6a3bc0 EFLAGS: 00010086 RAX: 0000000000000000 RBX: ffffb6bb9e6a3c80 RCX: ffff90ad0d209400 RDX: 0000000000000008 RSI: ffffb6bb9e6a3c80 RDI: ffff90eb3b236438 RBP: ffff90eb3b236438 R08: 0000005c743512ab R09: ffffffffffff0000 R10: 0000000000000001 R11: 0000000000000100 R12: ffff90eb3b236438 R13: ffff90eb3b2364f0 R14: ffff90eb3b6359c0 R15: ffff90eb3b6359c0 FS: 0000000000000000(0000) GS:ffff90eb3df00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000040 CR3: 000000807da3c006 CR4: 0000000000770ef0 PKRU: 55555554 Call Trace: <TASK> ? __die+0x24/0x70 ? page_fault_oops+0x14a/0x510 ? exc_page_fault+0x77/0x170 ? asm_exc_page_fault+0x26/0x30 ? can_migrate_task+0x2b/0x6c0 sched_balance_rq+0x7a8/0x1190 sched_balance_newidle+0x1e2/0x490 pick_next_task_fair+0x36/0x4a0 __schedule+0x1c0/0x1710 ? srso_alias_return_thunk+0x5/0xfbef5 ? refill_stock+0x1a/0x30 ? srso_alias_return_thunk+0x5/0xfbef5 ? obj_cgroup_uncharge_pages+0x4d/0xd0 do_task_dead+0x42/0x50 do_exit+0x777/0xad0 do_group_exit+0x30/0x80 __x64_sys_exit_group+0x18/0x20 do_syscall_64+0x79/0x120 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit_to_user_mode+0x5b/0x170 entry_SYSCALL_64_after_hwframe+0x6c/0x74 RIP: 0033:0x7f963a6eac31 Code: Unable to access opcode bytes at 0x7f963a6eac07. RSP: 002b:00007ffc6b7158c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 RAX: ffffffffffffffda RBX: 00007f963a816a00 RCX: 00007f963a6eac31 RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffffffffffffff80 R09: 0000000000000020 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f963a816a00 R13: 0000000000000000 R14: 00007f963a81bee8 R15: 00007f963a81bf00 </TASK> Modules linked in: ... CR2: 0000000000000040 ---[ end trace 0000000000000000 ]---
$ scripts/faddr2line vmlinux can_migrate_task+0x2b/0x6c0
can_migrate_task+0x2b/0x6c0:
throttled_lb_pair at kernel/sched/fair.c:5738
(inlined by) can_migrate_task at kernel/sched/fair.c:9090 Corresponds to:
static inline int throttled_lb_pair(struct task_group *tg,
int src_cpu, int dest_cpu)
{
struct cfs_rq *src_cfs_rq, *dest_cfs_rq;
src_cfs_rq = tg->cfs_rq[src_cpu]; /* <----- Here -----< */
dest_cfs_rq = tg->cfs_rq[dest_cpu];
return throttled_hierarchy(src_cfs_rq) ||
throttled_hierarchy(dest_cfs_rq);
}
(inlined by)
int can_migrate_task(struct task_struct *p, struct lb_env *env)
{
/* Called here */
if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
return 0;
...
}2. A NULL pointer dereferencing in pick_next_task_fair():
BUG: kernel NULL pointer dereference, address: 0000000000000098 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 107 PID: 1206665 Comm: spawn Tainted: G W 6.9.0-rc1-test+ #958 Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.7.3 03/30/2022 RIP: 0010:pick_next_task_fair+0x327/0x4a0 Code: ... RSP: 0018:ffffb613c212fd28 EFLAGS: 00010002 RAX: 0000004ed2799383 RBX: 0000000000000000 RCX: ffff8f65baf3f800 RDX: ffff8f65baf3ca00 RSI: 0000000000000000 RDI: 000000825ae302ab RBP: ffff8f64b13b59c0 R08: 0000000000000015 R09: 0000000000000314 R10: 0000000000000001 R11: 0000000000000001 R12: ffff8f261ac199c0 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 FS: 00007f768b79e740(0000) GS:ffff8f64b1380000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000098 CR3: 00000040d1010006 CR4: 0000000000770ef0 PKRU: 55555554 Call Trace: <TASK> ? __die+0x24/0x70 ? page_fault_oops+0x14a/0x510 ? srso_alias_return_thunk+0x5/0xfbef5 ? report_bug+0x18e/0x1a0 ? srso_alias_return_thunk+0x5/0xfbef5 ? exc_page_fault+0x77/0x170 ? asm_exc_page_fault+0x26/0x30 ? pick_next_task_fair+0x327/0x4a0 ? pick_next_task_fair+0x320/0x4a0 __schedule+0x1c0/0x1710 ? release_task+0x2fc/0x4c0 ? srso_alias_return_thunk+0x5/0xfbef5 schedule+0x30/0x120 syscall_exit_to_user_mode+0x98/0x1b0 do_syscall_64+0x85/0x120 ? srso_alias_return_thunk+0x5/0xfbef5 ? __count_memcg_events+0x69/0x100 ? srso_alias_return_thunk+0x5/0xfbef5 ? count_memcg_events.constprop.0+0x1a/0x30 ? srso_alias_return_thunk+0x5/0xfbef5 ? handle_mm_fault+0x17d/0x2e0 ? srso_alias_return_thunk+0x5/0xfbef5 ? do_user_addr_fault+0x33d/0x6f0 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit_to_user_mode+0x5b/0x170 entry_SYSCALL_64_after_hwframe+0x6c/0x74 RIP: 0033:0x7f768b4eab57 Code: ... RSP: 002b:00007fff5f6e2018 EFLAGS: 00000246 ORIG_RAX: 0000000000000038 RAX: 000000000026d13b RBX: 00007f768b7ee040 RCX: 00007f768b4eab57 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 00007f768b79ea10 R11: 0000000000000246 R12: 0000000000000001 R13: 000055b811d1a140 R14: 000055b811d1cd88 R15: 00007f768b7ee040 </TASK> Modules linked in: ... CR2: 0000000000000098 ---[ end trace 0000000000000000 ]---
$ scripts/faddr2line vmlinux pick_next_task_fair+0x327/0x4a0
pick_next_task_fair+0x327/0x4a0:
is_same_group at kernel/sched/fair.c:418
(inlined by) pick_next_task_fair at kernel/sched/fair.c:8625 struct cfs_rq *
is_same_group(struct sched_entity *se, struct sched_entity *pse)
{
if (se->cfs_rq == pse->cfs_rq) /* <----- HERE -----< */
return se->cfs_rq;
return NULL;
}
(inlined by)
struct task_struct *
pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
...
if (prev != p) {
...
while (!(cfs_rq = is_same_group(se, pse) /* <---- HERE ----< */)) {
...
}
...
}
...
}3. A NULL Pointer dereferencing in __dequeue_entity():
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 95 PID: 60896 Comm: spawn Not tainted 6.9.0-rc1-test+ #958 Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.7.3 03/30/2022 RIP: 0010:__rb_erase_color+0x88/0x260 Code: ... RSP: 0018:ffffab158755fc08 EFLAGS: 00010046 RAX: 0000000000000000 RBX: ffffffff841314b0 RCX: 0000000017fc8dd0 RDX: 0000000000000000 RSI: ffff8decfb1fe450 RDI: ffff8decf80bcdd0 RBP: ffff8decf80bcdd0 R08: ffff8decf80bcdd0 R09: ffffffffffffbb60 R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 R13: ffff8decfb1fe450 R14: ffff8ded0ec03400 R15: ffff8decfb1fe400 FS: 00007f1ded0a2740(0000) GS:ffff8e2bb0d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000040e66f0005 CR4: 0000000000770ef0 PKRU: 55555554 Call Trace: <TASK> ? __die+0x24/0x70 ? page_fault_oops+0x14a/0x510 ? exc_page_fault+0x77/0x170 ? asm_exc_page_fault+0x26/0x30 ? __pfx_min_vruntime_cb_rotate+0x10/0x10 ? __rb_erase_color+0x88/0x260 __dequeue_entity+0x1b7/0x310 set_next_entity+0xc0/0x1e0 pick_next_task_fair+0x355/0x4a0 __schedule+0x1c0/0x1710 ? native_queued_spin_lock_slowpath+0x2a4/0x2f0 schedule+0x30/0x120 do_wait+0xad/0x100 kernel_wait4+0xa9/0x150 ? __pfx_child_wait_callback+0x10/0x10 do_syscall_64+0x79/0x120 ? srso_alias_return_thunk+0x5/0xfbef5 ? __count_memcg_events+0x69/0x100 ? srso_alias_return_thunk+0x5/0xfbef5 ? count_memcg_events.constprop.0+0x1a/0x30 ? srso_alias_return_thunk+0x5/0xfbef5 ? handle_mm_fault+0x17d/0x2e0 ? srso_alias_return_thunk+0x5/0xfbef5 ? do_user_addr_fault+0x33d/0x6f0 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit_to_user_mode+0x5b/0x170 entry_SYSCALL_64_after_hwframe+0x6c/0x74 RIP: 0033:0x7f1deceea3ea Code: ... RSP: 002b:00007ffd7fd37ca8 EFLAGS: 00000246 ORIG_RAX: 000000000000003d RAX: ffffffffffffffda RBX: 00007ffd7fd37cb4 RCX: 00007f1deceea3ea RDX: 0000000000000000 RSI: 00007ffd7fd37cb4 RDI: 00000000ffffffff RBP: 0000000000000002 R08: 00000000000136f5 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd7fd37dd8 R13: 000055fd8debf140 R14: 000055fd8dec1d88 R15: 00007f1ded0f2040 </TASK> Modules linked in: ... CR2: 0000000000000000 ---[ end trace 0000000000000000 ]---
Note: I only ran into this issue with unixbench spawn. A bunch of other benchmarks (hackbench, stream, tbench, netperf, schbench, other variants of unixbench) ran fine without bringing down the machine.
Attaching my config below in case this in config specific.
Quoted 10 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/sched.h | 1 kernel/sched/core.c | 22 +++++-- kernel/sched/fair.c | 148 +++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/features.h | 12 +++ kernel/sched/sched.h | 2 5 files changed, 167 insertions(+), 18 deletions(-)
Quoted 2 lines
[..snip..]
--
Thanks and Regards,
Prateek
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 6.9.0-rc1 Kernel Configuration
#CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0" CONFIG_CC_IS_GCC=yCONFIG_GCC_VERSION=110400CONFIG_CLANG_VERSION=0CONFIG_AS_IS_GNU=yCONFIG_AS_VERSION=23800CONFIG_LD_IS_BFD=yCONFIG_LD_VERSION=23800CONFIG_LLD_VERSION=0CONFIG_CC_CAN_LINK=yCONFIG_CC_CAN_LINK_STATIC=y CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y CONFIG_GCC_ASM_GOTO_OUTPUT_WORKAROUND=y CONFIG_TOOLS_SUPPORT_RELR=yCONFIG_CC_HAS_ASM_INLINE=y CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y CONFIG_PAHOLE_VERSION=125CONFIG_IRQ_WORK=yCONFIG_BUILDTIME_TABLE_SORT=y CONFIG_THREAD_INFO_IN_TASK=y
#
# General setup
#CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
# CONFIG_WERROR is not set
# CONFIG_UAPI_HEADER_TEST is not setCONFIG_LOCALVERSION="-test"
# CONFIG_LOCALVERSION_AUTO is not setCONFIG_BUILD_SALT=""CONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_BZIP2=yCONFIG_HAVE_KERNEL_LZMA=yCONFIG_HAVE_KERNEL_XZ=yCONFIG_HAVE_KERNEL_LZO=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_HAVE_KERNEL_ZSTD=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not setCONFIG_KERNEL_ZSTD=yCONFIG_DEFAULT_INIT=""CONFIG_DEFAULT_HOSTNAME="(none)"CONFIG_SYSVIPC=yCONFIG_SYSVIPC_SYSCTL=yCONFIG_SYSVIPC_COMPAT=yCONFIG_POSIX_MQUEUE=yCONFIG_POSIX_MQUEUE_SYSCTL=yCONFIG_WATCH_QUEUE=yCONFIG_CROSS_MEMORY_ATTACH=yCONFIG_USELIB=yCONFIG_AUDIT=yCONFIG_HAVE_ARCH_AUDITSYSCALL=yCONFIG_AUDITSYSCALL=y
#
# IRQ subsystem
#CONFIG_GENERIC_IRQ_PROBE=yCONFIG_GENERIC_IRQ_SHOW=yCONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=yCONFIG_GENERIC_PENDING_IRQ=yCONFIG_GENERIC_IRQ_MIGRATION=yCONFIG_HARDIRQS_SW_RESEND=yCONFIG_GENERIC_IRQ_CHIP=yCONFIG_IRQ_DOMAIN=yCONFIG_IRQ_DOMAIN_HIERARCHY=yCONFIG_GENERIC_MSI_IRQ=yCONFIG_IRQ_MSI_IOMMU=yCONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR=yCONFIG_GENERIC_IRQ_RESERVATION_MODE=yCONFIG_IRQ_FORCED_THREADING=yCONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set
# end of IRQ subsystem
CONFIG_CLOCKSOURCE_WATCHDOG=yCONFIG_ARCH_CLOCKSOURCE_INIT=yCONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=yCONFIG_GENERIC_TIME_VSYSCALL=yCONFIG_GENERIC_CLOCKEVENTS=yCONFIG_GENERIC_CLOCKEVENTS_BROADCAST=yCONFIG_GENERIC_CLOCKEVENTS_BROADCAST_IDLE=yCONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=yCONFIG_GENERIC_CMOS_UPDATE=yCONFIG_HAVE_POSIX_CPU_TIMERS_TASK_WORK=yCONFIG_POSIX_CPU_TIMERS_TASK_WORK=yCONFIG_CONTEXT_TRACKING=yCONFIG_CONTEXT_TRACKING_IDLE=y
#
# Timers subsystem
#CONFIG_TICK_ONESHOT=yCONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not setCONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not setCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=yCONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=125
# end of Timers subsystem
CONFIG_BPF=yCONFIG_HAVE_EBPF_JIT=yCONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
#
# BPF subsystem
#CONFIG_BPF_SYSCALL=yCONFIG_BPF_JIT=yCONFIG_BPF_JIT_ALWAYS_ON=yCONFIG_BPF_JIT_DEFAULT_ON=yCONFIG_BPF_UNPRIV_DEFAULT_OFF=y
# CONFIG_BPF_PRELOAD is not setCONFIG_BPF_LSM=y
# end of BPF subsystem
CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not setCONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not setCONFIG_PREEMPT_COUNT=yCONFIG_PREEMPTION=yCONFIG_PREEMPT_DYNAMIC=yCONFIG_SCHED_CORE=y
#
# CPU/Task time and stats accounting
#CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not setCONFIG_BSD_PROCESS_ACCT=yCONFIG_BSD_PROCESS_ACCT_V3=yCONFIG_TASKSTATS=yCONFIG_TASK_DELAY_ACCT=yCONFIG_TASK_XACCT=yCONFIG_TASK_IO_ACCOUNTING=yCONFIG_PSI=y
# CONFIG_PSI_DEFAULT_DISABLED is not set
# end of CPU/Task time and stats accounting
CONFIG_CPU_ISOLATION=y
#
# RCU Subsystem
#CONFIG_TREE_RCU=yCONFIG_PREEMPT_RCU=y
# CONFIG_RCU_EXPERT is not setCONFIG_TREE_SRCU=yCONFIG_TASKS_RCU_GENERIC=yCONFIG_TASKS_RCU=yCONFIG_TASKS_RUDE_RCU=yCONFIG_TASKS_TRACE_RCU=yCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_NEED_SEGCBLIST=y
# end of RCU Subsystem
# CONFIG_IKCONFIG is not setCONFIG_IKHEADERS=mCONFIG_LOG_BUF_SHIFT=18CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
# CONFIG_PRINTK_INDEX is not setCONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
#
# Scheduler features
#CONFIG_UCLAMP_TASK=yCONFIG_UCLAMP_BUCKETS_COUNT=5
# end of Scheduler features
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y CONFIG_CC_HAS_INT128=y CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5" CONFIG_GCC10_NO_ARRAY_BOUNDS=y CONFIG_CC_NO_ARRAY_BOUNDS=y CONFIG_GCC_NO_STRINGOP_OVERFLOW=y CONFIG_CC_NO_STRINGOP_OVERFLOW=y CONFIG_ARCH_SUPPORTS_INT128=y CONFIG_NUMA_BALANCING=y CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y CONFIG_CGROUPS=yCONFIG_PAGE_COUNTER=y
# CONFIG_CGROUP_FAVOR_DYNMODS is not set CONFIG_MEMCG=yCONFIG_MEMCG_KMEM=yCONFIG_BLK_CGROUP=yCONFIG_CGROUP_WRITEBACK=yCONFIG_CGROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_CFS_BANDWIDTH=y
# CONFIG_RT_GROUP_SCHED is not set CONFIG_SCHED_MM_CID=yCONFIG_UCLAMP_TASK_GROUP=yCONFIG_CGROUP_PIDS=yCONFIG_CGROUP_RDMA=yCONFIG_CGROUP_FREEZER=yCONFIG_CGROUP_HUGETLB=yCONFIG_CPUSETS=yCONFIG_PROC_PID_CPUSET=yCONFIG_CGROUP_DEVICE=yCONFIG_CGROUP_CPUACCT=yCONFIG_CGROUP_PERF=yCONFIG_CGROUP_BPF=yCONFIG_CGROUP_MISC=y
# CONFIG_CGROUP_DEBUG is not set CONFIG_SOCK_CGROUP_DATA=yCONFIG_NAMESPACES=yCONFIG_UTS_NS=yCONFIG_TIME_NS=yCONFIG_IPC_NS=yCONFIG_USER_NS=yCONFIG_PID_NS=yCONFIG_NET_NS=yCONFIG_CHECKPOINT_RESTORE=y CONFIG_SCHED_AUTOGROUP=yCONFIG_RELAY=yCONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE=""CONFIG_RD_GZIP=yCONFIG_RD_BZIP2=yCONFIG_RD_LZMA=yCONFIG_RD_XZ=yCONFIG_RD_LZO=yCONFIG_RD_LZ4=yCONFIG_RD_ZSTD=yCONFIG_BOOT_CONFIG=y
# CONFIG_BOOT_CONFIG_FORCE is not set
# CONFIG_BOOT_CONFIG_EMBED is not set CONFIG_INITRAMFS_PRESERVE_MTIME=y CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set CONFIG_LD_ORPHAN_WARN=y CONFIG_LD_ORPHAN_WARN_LEVEL="warn" CONFIG_SYSCTL=yCONFIG_HAVE_UID16=y CONFIG_SYSCTL_EXCEPTION_TRACE=y CONFIG_HAVE_PCSPKR_PLATFORM=yCONFIG_EXPERT=yCONFIG_UID16=yCONFIG_MULTIUSER=yCONFIG_SGETMASK_SYSCALL=yCONFIG_SYSFS_SYSCALL=yCONFIG_FHANDLE=yCONFIG_POSIX_TIMERS=yCONFIG_PRINTK=yCONFIG_BUG=yCONFIG_ELF_CORE=yCONFIG_PCSPKR_PLATFORM=yCONFIG_BASE_FULL=yCONFIG_FUTEX=yCONFIG_FUTEX_PI=yCONFIG_EPOLL=yCONFIG_SIGNALFD=yCONFIG_TIMERFD=yCONFIG_EVENTFD=yCONFIG_SHMEM=yCONFIG_AIO=yCONFIG_IO_URING=yCONFIG_ADVISE_SYSCALLS=yCONFIG_MEMBARRIER=yCONFIG_KCMP=yCONFIG_RSEQ=y
# CONFIG_DEBUG_RSEQ is not set CONFIG_CACHESTAT_SYSCALL=yCONFIG_PC104=yCONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_SELFTEST is not set CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y CONFIG_KALLSYMS_BASE_RELATIVE=y CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_GUEST_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
# end of Kernel Performance Events And Counters
CONFIG_SYSTEM_DATA_VERIFICATION=yCONFIG_PROFILING=yCONFIG_TRACEPOINTS=y
#
# Kexec and crash features
#CONFIG_CRASH_RESERVE=yCONFIG_VMCORE_INFO=yCONFIG_KEXEC_CORE=yCONFIG_HAVE_IMA_KEXEC=yCONFIG_KEXEC=yCONFIG_KEXEC_FILE=yCONFIG_KEXEC_SIG=y
# CONFIG_KEXEC_SIG_FORCE is not setCONFIG_KEXEC_BZIMAGE_VERIFY_SIG=yCONFIG_KEXEC_JUMP=yCONFIG_CRASH_DUMP=yCONFIG_CRASH_HOTPLUG=yCONFIG_CRASH_MAX_MEMORY_RANGES=8192
# end of Kexec and crash features # end of General setup
CONFIG_64BIT=yCONFIG_X86_64=yCONFIG_X86=yCONFIG_INSTRUCTION_DECODER=yCONFIG_OUTPUT_FORMAT="elf64-x86-64"CONFIG_LOCKDEP_SUPPORT=yCONFIG_STACKTRACE_SUPPORT=yCONFIG_MMU=yCONFIG_ARCH_MMAP_RND_BITS_MIN=28CONFIG_ARCH_MMAP_RND_BITS_MAX=32CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16CONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_BUG=yCONFIG_GENERIC_BUG_RELATIVE_POINTERS=yCONFIG_ARCH_MAY_HAVE_PC_FDC=yCONFIG_GENERIC_CALIBRATE_DELAY=yCONFIG_ARCH_HAS_CPU_RELAX=yCONFIG_ARCH_HIBERNATION_POSSIBLE=yCONFIG_ARCH_SUSPEND_POSSIBLE=yCONFIG_AUDIT_ARCH=yCONFIG_HAVE_INTEL_TXT=yCONFIG_X86_64_SMP=yCONFIG_ARCH_SUPPORTS_UPROBES=yCONFIG_FIX_EARLYCON_MEM=yCONFIG_DYNAMIC_PHYSICAL_MASK=yCONFIG_PGTABLE_LEVELS=5CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
#
# Processor type and features
#CONFIG_SMP=yCONFIG_X86_X2APIC=yCONFIG_X86_MPPARSE=yCONFIG_X86_CPU_RESCTRL=y
# CONFIG_X86_FRED is not setCONFIG_X86_EXTENDED_PLATFORM=yCONFIG_X86_NUMACHIP=y
# CONFIG_X86_VSMP is not setCONFIG_X86_UV=y
# CONFIG_X86_GOLDFISH is not set
# CONFIG_X86_INTEL_MID is not setCONFIG_X86_INTEL_LPSS=yCONFIG_X86_AMD_PLATFORM_DEVICE=yCONFIG_IOSF_MBI=yCONFIG_IOSF_MBI_DEBUG=yCONFIG_X86_SUPPORTS_MEMORY_FAILURE=yCONFIG_SCHED_OMIT_FRAME_POINTER=yCONFIG_HYPERVISOR_GUEST=yCONFIG_PARAVIRT=yCONFIG_PARAVIRT_XXL=y
# CONFIG_PARAVIRT_DEBUG is not setCONFIG_PARAVIRT_SPINLOCKS=yCONFIG_X86_HV_CALLBACK_VECTOR=yCONFIG_XEN=yCONFIG_XEN_PV=yCONFIG_XEN_512GB=yCONFIG_XEN_PV_SMP=yCONFIG_XEN_PV_DOM0=yCONFIG_XEN_PVHVM=yCONFIG_XEN_PVHVM_SMP=yCONFIG_XEN_PVHVM_GUEST=yCONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not setCONFIG_XEN_PVH=yCONFIG_XEN_DOM0=yCONFIG_XEN_PV_MSR_SAFE=yCONFIG_KVM_GUEST=yCONFIG_ARCH_CPUIDLE_HALTPOLL=yCONFIG_PVH=y
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not setCONFIG_PARAVIRT_CLOCK=yCONFIG_JAILHOUSE_GUEST=yCONFIG_ACRN_GUEST=y
# CONFIG_INTEL_TDX_GUEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not setCONFIG_GENERIC_CPU=yCONFIG_X86_INTERNODE_CACHE_SHIFT=6CONFIG_X86_L1_CACHE_SHIFT=6CONFIG_X86_TSC=yCONFIG_X86_HAVE_PAE=yCONFIG_X86_CMPXCHG64=yCONFIG_X86_CMOV=yCONFIG_X86_MINIMUM_CPU_FAMILY=64CONFIG_X86_DEBUGCTLMSR=yCONFIG_IA32_FEAT_CTL=yCONFIG_X86_VMX_FEATURE_NAMES=yCONFIG_PROCESSOR_SELECT=yCONFIG_CPU_SUP_INTEL=yCONFIG_CPU_SUP_AMD=yCONFIG_CPU_SUP_HYGON=yCONFIG_CPU_SUP_CENTAUR=yCONFIG_CPU_SUP_ZHAOXIN=yCONFIG_HPET_TIMER=yCONFIG_HPET_EMULATE_RTC=yCONFIG_DMI=yCONFIG_GART_IOMMU=yCONFIG_BOOT_VESA_SUPPORT=yCONFIG_MAXSMP=yCONFIG_NR_CPUS_RANGE_BEGIN=8192CONFIG_NR_CPUS_RANGE_END=8192CONFIG_NR_CPUS_DEFAULT=8192CONFIG_NR_CPUS=8192CONFIG_SCHED_CLUSTER=yCONFIG_SCHED_SMT=yCONFIG_SCHED_MC=yCONFIG_SCHED_MC_PRIO=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=yCONFIG_X86_MCE=yCONFIG_X86_MCELOG_LEGACY=yCONFIG_X86_MCE_INTEL=yCONFIG_X86_MCE_AMD=yCONFIG_X86_MCE_THRESHOLD=yCONFIG_X86_MCE_INJECT=m
#
# Performance monitoring
#CONFIG_PERF_EVENTS_INTEL_UNCORE=yCONFIG_PERF_EVENTS_INTEL_RAPL=mCONFIG_PERF_EVENTS_INTEL_CSTATE=m
# CONFIG_PERF_EVENTS_AMD_POWER is not setCONFIG_PERF_EVENTS_AMD_UNCORE=m
# CONFIG_PERF_EVENTS_AMD_BRS is not set
# end of Performance monitoring
CONFIG_X86_16BIT=yCONFIG_X86_ESPFIX64=yCONFIG_X86_VSYSCALL_EMULATION=yCONFIG_X86_IOPL_IOPERM=yCONFIG_MICROCODE=yCONFIG_MICROCODE_LATE_LOADING=y
# CONFIG_MICROCODE_LATE_FORCE_MINREV is not setCONFIG_X86_MSR=mCONFIG_X86_CPUID=mCONFIG_X86_5LEVEL=yCONFIG_X86_DIRECT_GBPAGES=y
# CONFIG_X86_CPA_STATISTICS is not setCONFIG_X86_MEM_ENCRYPT=yCONFIG_AMD_MEM_ENCRYPT=yCONFIG_NUMA=yCONFIG_AMD_NUMA=yCONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not setCONFIG_NODES_SHIFT=10CONFIG_ARCH_SPARSEMEM_ENABLE=yCONFIG_ARCH_SPARSEMEM_DEFAULT=yCONFIG_ARCH_MEMORY_PROBE=yCONFIG_ARCH_PROC_KCORE_TEXT=yCONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000CONFIG_X86_PMEM_LEGACY_DEVICE=yCONFIG_X86_PMEM_LEGACY=yCONFIG_X86_CHECK_BIOS_CORRUPTION=yCONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=yCONFIG_MTRR=yCONFIG_MTRR_SANITIZER=yCONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1CONFIG_X86_PAT=yCONFIG_ARCH_USES_PG_UNCACHED=yCONFIG_X86_UMIP=yCONFIG_CC_HAS_IBT=yCONFIG_X86_CET=yCONFIG_X86_KERNEL_IBT=yCONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=yCONFIG_X86_INTEL_TSX_MODE_OFF=y
# CONFIG_X86_INTEL_TSX_MODE_ON is not set
# CONFIG_X86_INTEL_TSX_MODE_AUTO is not setCONFIG_X86_SGX=y
# CONFIG_X86_USER_SHADOW_STACK is not setCONFIG_EFI=yCONFIG_EFI_STUB=yCONFIG_EFI_HANDOVER_PROTOCOL=yCONFIG_EFI_MIXED=y
# CONFIG_EFI_FAKE_MEMMAP is not setCONFIG_EFI_RUNTIME_MAP=y
# CONFIG_HZ_100 is not setCONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not setCONFIG_HZ=250CONFIG_SCHED_HRTICK=yCONFIG_ARCH_SUPPORTS_KEXEC=yCONFIG_ARCH_SUPPORTS_KEXEC_FILE=yCONFIG_ARCH_SELECTS_KEXEC_FILE=yCONFIG_ARCH_SUPPORTS_KEXEC_PURGATORY=yCONFIG_ARCH_SUPPORTS_KEXEC_SIG=yCONFIG_ARCH_SUPPORTS_KEXEC_SIG_FORCE=yCONFIG_ARCH_SUPPORTS_KEXEC_BZIMAGE_VERIFY_SIG=yCONFIG_ARCH_SUPPORTS_KEXEC_JUMP=yCONFIG_ARCH_SUPPORTS_CRASH_DUMP=yCONFIG_ARCH_SUPPORTS_CRASH_HOTPLUG=yCONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION=yCONFIG_PHYSICAL_START=0x1000000CONFIG_RELOCATABLE=yCONFIG_RANDOMIZE_BASE=yCONFIG_X86_NEED_RELOCS=yCONFIG_PHYSICAL_ALIGN=0x200000CONFIG_DYNAMIC_MEMORY_LAYOUT=yCONFIG_RANDOMIZE_MEMORY=yCONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
# CONFIG_ADDRESS_MASKING is not setCONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not setCONFIG_LEGACY_VSYSCALL_XONLY=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not setCONFIG_MODIFY_LDT_SYSCALL=y
# CONFIG_STRICT_SIGALTSTACK_SIZE is not setCONFIG_HAVE_LIVEPATCH=yCONFIG_LIVEPATCH=y
# end of Processor type and features
CONFIG_CC_HAS_SLS=yCONFIG_CC_HAS_RETURN_THUNK=yCONFIG_CC_HAS_ENTRY_PADDING=yCONFIG_FUNCTION_PADDING_CFI=11CONFIG_FUNCTION_PADDING_BYTES=16CONFIG_CALL_PADDING=yCONFIG_HAVE_CALL_THUNKS=yCONFIG_CALL_THUNKS=yCONFIG_PREFIX_SYMBOLS=yCONFIG_SPECULATION_MITIGATIONS=yCONFIG_MITIGATION_PAGE_TABLE_ISOLATION=yCONFIG_MITIGATION_RETPOLINE=yCONFIG_MITIGATION_RETHUNK=yCONFIG_MITIGATION_UNRET_ENTRY=yCONFIG_MITIGATION_CALL_DEPTH_TRACKING=y
# CONFIG_CALL_THUNKS_DEBUG is not setCONFIG_MITIGATION_IBPB_ENTRY=yCONFIG_MITIGATION_IBRS_ENTRY=yCONFIG_MITIGATION_SRSO=y
# CONFIG_MITIGATION_SLS is not set
# CONFIG_MITIGATION_GDS_FORCE is not setCONFIG_MITIGATION_RFDS=yCONFIG_ARCH_HAS_ADD_PAGES=y
#
# Power management and ACPI options
#CONFIG_ARCH_HIBERNATION_HEADER=yCONFIG_SUSPEND=yCONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not setCONFIG_HIBERNATE_CALLBACKS=yCONFIG_HIBERNATION=yCONFIG_HIBERNATION_SNAPSHOT_DEV=yCONFIG_HIBERNATION_COMP_LZO=y
# CONFIG_HIBERNATION_COMP_LZ4 is not setCONFIG_HIBERNATION_DEF_COMP="lzo"CONFIG_PM_STD_PARTITION=""CONFIG_PM_SLEEP=yCONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_USERSPACE_AUTOSLEEP is not setCONFIG_PM_WAKELOCKS=yCONFIG_PM_WAKELOCKS_LIMIT=100CONFIG_PM_WAKELOCKS_GC=yCONFIG_PM=yCONFIG_PM_DEBUG=yCONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_TEST_SUSPEND is not setCONFIG_PM_SLEEP_DEBUG=y
# CONFIG_DPM_WATCHDOG is not setCONFIG_PM_TRACE=yCONFIG_PM_TRACE_RTC=yCONFIG_PM_CLK=yCONFIG_PM_GENERIC_DOMAINS=yCONFIG_WQ_POWER_EFFICIENT_DEFAULT=yCONFIG_PM_GENERIC_DOMAINS_SLEEP=yCONFIG_ENERGY_MODEL=yCONFIG_ARCH_SUPPORTS_ACPI=yCONFIG_ACPI=yCONFIG_ACPI_LEGACY_TABLES_LOOKUP=yCONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=yCONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=yCONFIG_ACPI_TABLE_LIB=yCONFIG_ACPI_THERMAL_LIB=yCONFIG_ACPI_DEBUGGER=yCONFIG_ACPI_DEBUGGER_USER=yCONFIG_ACPI_SPCR_TABLE=yCONFIG_ACPI_FPDT=yCONFIG_ACPI_LPIT=yCONFIG_ACPI_SLEEP=yCONFIG_ACPI_REV_OVERRIDE_POSSIBLE=yCONFIG_ACPI_EC_DEBUGFS=mCONFIG_ACPI_AC=yCONFIG_ACPI_BATTERY=yCONFIG_ACPI_BUTTON=yCONFIG_ACPI_VIDEO=mCONFIG_ACPI_FAN=yCONFIG_ACPI_TAD=mCONFIG_ACPI_DOCK=yCONFIG_ACPI_CPU_FREQ_PSS=yCONFIG_ACPI_PROCESSOR_CSTATE=yCONFIG_ACPI_PROCESSOR_IDLE=yCONFIG_ACPI_CPPC_LIB=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_IPMI=mCONFIG_ACPI_HOTPLUG_CPU=yCONFIG_ACPI_PROCESSOR_AGGREGATOR=mCONFIG_ACPI_THERMAL=yCONFIG_ACPI_PLATFORM_PROFILE=mCONFIG_ACPI_CUSTOM_DSDT_FILE=""CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=yCONFIG_ACPI_TABLE_UPGRADE=yCONFIG_ACPI_DEBUG=yCONFIG_ACPI_PCI_SLOT=yCONFIG_ACPI_CONTAINER=yCONFIG_ACPI_HOTPLUG_MEMORY=yCONFIG_ACPI_HOTPLUG_IOAPIC=yCONFIG_ACPI_SBS=mCONFIG_ACPI_HED=yCONFIG_ACPI_BGRT=y
# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not setCONFIG_ACPI_NFIT=m
# CONFIG_NFIT_SECURITY_DEBUG is not setCONFIG_ACPI_NUMA=yCONFIG_ACPI_HMAT=yCONFIG_HAVE_ACPI_APEI=yCONFIG_HAVE_ACPI_APEI_NMI=yCONFIG_ACPI_APEI=yCONFIG_ACPI_APEI_GHES=yCONFIG_ACPI_APEI_PCIEAER=yCONFIG_ACPI_APEI_MEMORY_FAILURE=yCONFIG_ACPI_APEI_EINJ=mCONFIG_ACPI_APEI_EINJ_CXL=y
# CONFIG_ACPI_APEI_ERST_DEBUG is not setCONFIG_ACPI_DPTF=yCONFIG_DPTF_POWER=mCONFIG_DPTF_PCH_FIVR=mCONFIG_ACPI_WATCHDOG=yCONFIG_ACPI_EXTLOG=mCONFIG_ACPI_ADXL=yCONFIG_ACPI_CONFIGFS=m
# CONFIG_ACPI_PFRUT is not setCONFIG_ACPI_PCC=y
# CONFIG_ACPI_FFH is not setCONFIG_PMIC_OPREGION=yCONFIG_BYTCRC_PMIC_OPREGION=yCONFIG_CHTCRC_PMIC_OPREGION=yCONFIG_XPOWER_PMIC_OPREGION=yCONFIG_BXT_WC_PMIC_OPREGION=yCONFIG_CHT_WC_PMIC_OPREGION=yCONFIG_CHT_DC_TI_PMIC_OPREGION=yCONFIG_TPS68470_PMIC_OPREGION=yCONFIG_ACPI_VIOT=yCONFIG_ACPI_PRMT=yCONFIG_X86_PM_TIMER=y
#
# CPU Frequency scaling
#CONFIG_CPU_FREQ=y CONFIG_CPU_FREQ_GOV_ATTR_SET=y CONFIG_CPU_FREQ_GOV_COMMON=y CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL=y CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
#
# CPU frequency scaling drivers
#CONFIG_X86_INTEL_PSTATE=yCONFIG_X86_PCC_CPUFREQ=yCONFIG_X86_AMD_PSTATE=yCONFIG_X86_AMD_PSTATE_DEFAULT_MODE=3
# CONFIG_X86_AMD_PSTATE_UT is not setCONFIG_X86_ACPI_CPUFREQ=yCONFIG_X86_ACPI_CPUFREQ_CPB=yCONFIG_X86_POWERNOW_K8=yCONFIG_X86_AMD_FREQ_SENSITIVITY=mCONFIG_X86_SPEEDSTEP_CENTRINO=yCONFIG_X86_P4_CLOCKMOD=m
#
# shared options # CONFIG_X86_SPEEDSTEP_LIB=m # end of CPU Frequency scaling
#
# CPU Idle
#CONFIG_CPU_IDLE=yCONFIG_CPU_IDLE_GOV_LADDER=yCONFIG_CPU_IDLE_GOV_MENU=yCONFIG_CPU_IDLE_GOV_TEO=yCONFIG_CPU_IDLE_GOV_HALTPOLL=yCONFIG_HALTPOLL_CPUIDLE=m
# end of CPU Idle
CONFIG_INTEL_IDLE=y
# end of Power management and ACPI options
#
# Bus options (PCI etc.)
#CONFIG_PCI_DIRECT=yCONFIG_PCI_MMCONFIG=yCONFIG_PCI_XEN=yCONFIG_MMCONF_FAM10H=y
# CONFIG_PCI_CNB20LE_QUIRK is not setCONFIG_ISA_BUS=yCONFIG_ISA_DMA_API=yCONFIG_AMD_NB=y
# end of Bus options (PCI etc.)
#
# Binary Emulations
#CONFIG_IA32_EMULATION=y
# CONFIG_IA32_EMULATION_DEFAULT_DISABLED is not set # CONFIG_X86_X32_ABI is not set CONFIG_COMPAT_32=yCONFIG_COMPAT=y CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
# end of Binary Emulations
CONFIG_KVM_COMMON=yCONFIG_HAVE_KVM_PFNCACHE=yCONFIG_HAVE_KVM_IRQCHIP=yCONFIG_HAVE_KVM_IRQ_ROUTING=yCONFIG_HAVE_KVM_DIRTY_RING=yCONFIG_HAVE_KVM_DIRTY_RING_TSO=yCONFIG_HAVE_KVM_DIRTY_RING_ACQ_REL=yCONFIG_KVM_MMIO=yCONFIG_KVM_ASYNC_PF=yCONFIG_HAVE_KVM_MSI=yCONFIG_HAVE_KVM_READONLY_MEM=yCONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=yCONFIG_KVM_VFIO=yCONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=yCONFIG_KVM_COMPAT=yCONFIG_HAVE_KVM_IRQ_BYPASS=yCONFIG_HAVE_KVM_NO_POLL=yCONFIG_KVM_XFER_TO_GUEST_WORK=yCONFIG_HAVE_KVM_PM_NOTIFIER=yCONFIG_KVM_GENERIC_HARDWARE_ENABLING=yCONFIG_KVM_GENERIC_MMU_NOTIFIER=yCONFIG_VIRTUALIZATION=yCONFIG_KVM=mCONFIG_KVM_WERROR=y
# CONFIG_KVM_SW_PROTECTED_VM is not setCONFIG_KVM_INTEL=mCONFIG_X86_SGX_KVM=yCONFIG_KVM_AMD=mCONFIG_KVM_AMD_SEV=yCONFIG_KVM_SMM=yCONFIG_KVM_HYPERV=yCONFIG_KVM_XEN=y
# CONFIG_KVM_PROVE_MMU is not setCONFIG_KVM_EXTERNAL_WRITE_TRACKING=yCONFIG_KVM_MAX_NR_VCPUS=4096CONFIG_AS_AVX512=yCONFIG_AS_SHA1_NI=yCONFIG_AS_SHA256_NI=yCONFIG_AS_TPAUSE=yCONFIG_AS_GFNI=yCONFIG_AS_WRUSS=y
#
# General architecture-dependent options
#CONFIG_HOTPLUG_SMT=yCONFIG_HOTPLUG_CORE_SYNC=yCONFIG_HOTPLUG_CORE_SYNC_DEAD=yCONFIG_HOTPLUG_CORE_SYNC_FULL=yCONFIG_HOTPLUG_SPLIT_STARTUP=yCONFIG_HOTPLUG_PARALLEL=yCONFIG_GENERIC_ENTRY=yCONFIG_KPROBES=yCONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
# CONFIG_STATIC_CALL_SELFTEST is not setCONFIG_OPTPROBES=yCONFIG_KPROBES_ON_FTRACE=yCONFIG_UPROBES=yCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=yCONFIG_ARCH_USE_BUILTIN_BSWAP=yCONFIG_KRETPROBES=yCONFIG_KRETPROBE_ON_RETHOOK=yCONFIG_USER_RETURN_NOTIFIER=yCONFIG_HAVE_IOREMAP_PROT=yCONFIG_HAVE_KPROBES=yCONFIG_HAVE_KRETPROBES=yCONFIG_HAVE_OPTPROBES=yCONFIG_HAVE_KPROBES_ON_FTRACE=yCONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=yCONFIG_HAVE_FUNCTION_ERROR_INJECTION=yCONFIG_HAVE_NMI=yCONFIG_TRACE_IRQFLAGS_SUPPORT=yCONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=yCONFIG_HAVE_ARCH_TRACEHOOK=yCONFIG_HAVE_DMA_CONTIGUOUS=yCONFIG_GENERIC_SMP_IDLE_THREAD=yCONFIG_ARCH_HAS_FORTIFY_SOURCE=yCONFIG_ARCH_HAS_SET_MEMORY=yCONFIG_ARCH_HAS_SET_DIRECT_MAP=yCONFIG_ARCH_HAS_CPU_FINALIZE_INIT=yCONFIG_ARCH_HAS_CPU_PASID=yCONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=yCONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=yCONFIG_ARCH_WANTS_NO_INSTR=yCONFIG_HAVE_ASM_MODVERSIONS=yCONFIG_HAVE_REGS_AND_STACK_ACCESS_API=yCONFIG_HAVE_RSEQ=yCONFIG_HAVE_RUST=yCONFIG_HAVE_FUNCTION_ARG_ACCESS_API=yCONFIG_HAVE_HW_BREAKPOINT=yCONFIG_HAVE_MIXED_BREAKPOINTS_REGS=yCONFIG_HAVE_USER_RETURN_NOTIFIER=yCONFIG_HAVE_PERF_EVENTS_NMI=yCONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=yCONFIG_HAVE_PERF_REGS=yCONFIG_HAVE_PERF_USER_STACK_DUMP=yCONFIG_HAVE_ARCH_JUMP_LABEL=yCONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=yCONFIG_MMU_GATHER_TABLE_FREE=yCONFIG_MMU_GATHER_RCU_TABLE_FREE=yCONFIG_MMU_GATHER_MERGE_VMAS=yCONFIG_MMU_LAZY_TLB_REFCOUNT=yCONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=yCONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS=yCONFIG_HAVE_ALIGNED_STRUCT_PAGE=yCONFIG_HAVE_CMPXCHG_LOCAL=yCONFIG_HAVE_CMPXCHG_DOUBLE=yCONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=yCONFIG_ARCH_WANT_OLD_COMPAT_IPC=yCONFIG_HAVE_ARCH_SECCOMP=yCONFIG_HAVE_ARCH_SECCOMP_FILTER=yCONFIG_SECCOMP=yCONFIG_SECCOMP_FILTER=y
# CONFIG_SECCOMP_CACHE_DEBUG is not setCONFIG_HAVE_ARCH_STACKLEAK=yCONFIG_HAVE_STACKPROTECTOR=yCONFIG_STACKPROTECTOR=yCONFIG_STACKPROTECTOR_STRONG=yCONFIG_ARCH_SUPPORTS_LTO_CLANG=yCONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=yCONFIG_LTO_NONE=yCONFIG_ARCH_SUPPORTS_CFI_CLANG=yCONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=yCONFIG_HAVE_CONTEXT_TRACKING_USER=yCONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK=yCONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=yCONFIG_HAVE_IRQ_TIME_ACCOUNTING=yCONFIG_HAVE_MOVE_PUD=yCONFIG_HAVE_MOVE_PMD=yCONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=yCONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=yCONFIG_HAVE_ARCH_HUGE_VMAP=yCONFIG_HAVE_ARCH_HUGE_VMALLOC=yCONFIG_ARCH_WANT_HUGE_PMD_SHARE=yCONFIG_ARCH_WANT_PMD_MKWRITE=yCONFIG_HAVE_ARCH_SOFT_DIRTY=yCONFIG_HAVE_MOD_ARCH_SPECIFIC=yCONFIG_MODULES_USE_ELF_RELA=yCONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=yCONFIG_HAVE_SOFTIRQ_ON_OWN_STACK=yCONFIG_SOFTIRQ_ON_OWN_STACK=yCONFIG_ARCH_HAS_ELF_RANDOMIZE=yCONFIG_HAVE_ARCH_MMAP_RND_BITS=yCONFIG_HAVE_EXIT_THREAD=yCONFIG_ARCH_MMAP_RND_BITS=28CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=yCONFIG_ARCH_MMAP_RND_COMPAT_BITS=8CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=yCONFIG_HAVE_PAGE_SIZE_4KB=yCONFIG_PAGE_SIZE_4KB=yCONFIG_PAGE_SIZE_LESS_THAN_64KB=yCONFIG_PAGE_SIZE_LESS_THAN_256KB=yCONFIG_PAGE_SHIFT=12CONFIG_HAVE_OBJTOOL=yCONFIG_HAVE_JUMP_LABEL_HACK=yCONFIG_HAVE_NOINSTR_HACK=yCONFIG_HAVE_NOINSTR_VALIDATION=yCONFIG_HAVE_UACCESS_VALIDATION=yCONFIG_HAVE_STACK_VALIDATION=yCONFIG_HAVE_RELIABLE_STACKTRACE=yCONFIG_ISA_BUS_API=yCONFIG_OLD_SIGSUSPEND3=yCONFIG_COMPAT_OLD_SIGACTION=yCONFIG_COMPAT_32BIT_TIME=yCONFIG_HAVE_ARCH_VMAP_STACK=yCONFIG_VMAP_STACK=yCONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=yCONFIG_RANDOMIZE_KSTACK_OFFSET=yCONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=yCONFIG_ARCH_HAS_STRICT_KERNEL_RWX=yCONFIG_STRICT_KERNEL_RWX=yCONFIG_ARCH_HAS_STRICT_MODULE_RWX=yCONFIG_STRICT_MODULE_RWX=yCONFIG_HAVE_ARCH_PREL32_RELOCATIONS=yCONFIG_ARCH_USE_MEMREMAP_PROT=y
# CONFIG_LOCK_EVENT_COUNTS is not setCONFIG_ARCH_HAS_MEM_ENCRYPT=yCONFIG_ARCH_HAS_CC_PLATFORM=yCONFIG_HAVE_STATIC_CALL=yCONFIG_HAVE_STATIC_CALL_INLINE=yCONFIG_HAVE_PREEMPT_DYNAMIC=yCONFIG_HAVE_PREEMPT_DYNAMIC_CALL=yCONFIG_ARCH_WANT_LD_ORPHAN_WARN=yCONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=yCONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=yCONFIG_ARCH_HAS_ELFCORE_COMPAT=yCONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=yCONFIG_DYNAMIC_SIGFRAME=yCONFIG_HAVE_ARCH_NODE_DEV_GROUP=yCONFIG_ARCH_HAS_HW_PTE_YOUNG=yCONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not setCONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling
CONFIG_HAVE_GCC_PLUGINS=yCONFIG_FUNCTION_ALIGNMENT_4B=yCONFIG_FUNCTION_ALIGNMENT_16B=yCONFIG_FUNCTION_ALIGNMENT=16
# end of General architecture-dependent options
CONFIG_RT_MUTEXES=yCONFIG_BASE_SMALL=0CONFIG_MODULE_SIG_FORMAT=yCONFIG_MODULES=y
# CONFIG_MODULE_DEBUG is not set
# CONFIG_MODULE_FORCE_LOAD is not set CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODULE_UNLOAD_TAINT_TRACKING is not set CONFIG_MODVERSIONS=yCONFIG_ASM_MODVERSIONS=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set CONFIG_MODULE_SIG_ALL=y
# CONFIG_MODULE_SIG_SHA256 is not set
# CONFIG_MODULE_SIG_SHA384 is not set CONFIG_MODULE_SIG_SHA512=y
# CONFIG_MODULE_SIG_SHA3_256 is not set
# CONFIG_MODULE_SIG_SHA3_384 is not set
# CONFIG_MODULE_SIG_SHA3_512 is not set CONFIG_MODULE_SIG_HASH="sha512" CONFIG_MODULE_COMPRESS_NONE=y
# CONFIG_MODULE_COMPRESS_GZIP is not set
# CONFIG_MODULE_COMPRESS_XZ is not set
# CONFIG_MODULE_COMPRESS_ZSTD is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set CONFIG_MODPROBE_PATH="/sbin/modprobe"
# CONFIG_TRIM_UNUSED_KSYMS is not set CONFIG_MODULES_TREE_LOOKUP=yCONFIG_BLOCK=yCONFIG_BLOCK_LEGACY_AUTOLOAD=y CONFIG_BLK_RQ_ALLOC_TIME=y CONFIG_BLK_CGROUP_RWSTAT=y CONFIG_BLK_CGROUP_PUNT_BIO=y CONFIG_BLK_DEV_BSG_COMMON=yCONFIG_BLK_ICQ=yCONFIG_BLK_DEV_BSGLIB=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_INTEGRITY_T10=y CONFIG_BLK_DEV_WRITE_MOUNTED=y CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_DEV_THROTTLING_LOW is not set CONFIG_BLK_WBT=yCONFIG_BLK_WBT_MQ=y
# CONFIG_BLK_CGROUP_IOLATENCY is not set CONFIG_BLK_CGROUP_FC_APPID=y CONFIG_BLK_CGROUP_IOCOST=y CONFIG_BLK_CGROUP_IOPRIO=yCONFIG_BLK_DEBUG_FS=yCONFIG_BLK_DEBUG_FS_ZONED=yCONFIG_BLK_SED_OPAL=y CONFIG_BLK_INLINE_ENCRYPTION=y CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK=y
#
# Partition Types
#CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not setCONFIG_AIX_PARTITION=yCONFIG_OSF_PARTITION=yCONFIG_AMIGA_PARTITION=yCONFIG_ATARI_PARTITION=yCONFIG_MAC_PARTITION=yCONFIG_MSDOS_PARTITION=yCONFIG_BSD_DISKLABEL=yCONFIG_MINIX_SUBPARTITION=yCONFIG_SOLARIS_X86_PARTITION=yCONFIG_UNIXWARE_DISKLABEL=yCONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not setCONFIG_SGI_PARTITION=yCONFIG_ULTRIX_PARTITION=yCONFIG_SUN_PARTITION=yCONFIG_KARMA_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_SYSV68_PARTITION=yCONFIG_CMDLINE_PARTITION=y
# end of Partition Types
CONFIG_BLK_MQ_PCI=yCONFIG_BLK_MQ_VIRTIO=yCONFIG_BLK_PM=yCONFIG_BLOCK_HOLDER_DEPRECATED=yCONFIG_BLK_MQ_STACKING=y
#
# IO Schedulers
#CONFIG_MQ_IOSCHED_DEADLINE=yCONFIG_MQ_IOSCHED_KYBER=mCONFIG_IOSCHED_BFQ=mCONFIG_BFQ_GROUP_IOSCHED=y
# CONFIG_BFQ_CGROUP_DEBUG is not set
# end of IO Schedulers
CONFIG_PREEMPT_NOTIFIERS=yCONFIG_PADATA=yCONFIG_ASN1=yCONFIG_UNINLINE_SPIN_UNLOCK=yCONFIG_ARCH_SUPPORTS_ATOMIC_RMW=yCONFIG_MUTEX_SPIN_ON_OWNER=yCONFIG_RWSEM_SPIN_ON_OWNER=yCONFIG_LOCK_SPIN_ON_OWNER=yCONFIG_ARCH_USE_QUEUED_SPINLOCKS=yCONFIG_QUEUED_SPINLOCKS=yCONFIG_ARCH_USE_QUEUED_RWLOCKS=yCONFIG_QUEUED_RWLOCKS=yCONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=yCONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE=yCONFIG_ARCH_HAS_SYSCALL_WRAPPER=yCONFIG_FREEZER=y
#
# Executable file formats
#CONFIG_BINFMT_ELF=yCONFIG_COMPAT_BINFMT_ELF=yCONFIG_ELFCORE=yCONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=yCONFIG_BINFMT_SCRIPT=yCONFIG_BINFMT_MISC=mCONFIG_COREDUMP=y
# end of Executable file formats
#
# Memory Management options
#CONFIG_ZPOOL=yCONFIG_SWAP=yCONFIG_ZSWAP=y
# CONFIG_ZSWAP_DEFAULT_ON is not set
# CONFIG_ZSWAP_SHRINKER_DEFAULT_ON is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo" CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set # CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud" CONFIG_ZBUD=yCONFIG_Z3FOLD=mCONFIG_ZSMALLOC=y
# CONFIG_ZSMALLOC_STAT is not set CONFIG_ZSMALLOC_CHAIN_SIZE=8
#
# Slab allocator options
#CONFIG_SLUB=y
# CONFIG_SLUB_TINY is not setCONFIG_SLAB_MERGE_DEFAULT=yCONFIG_SLAB_FREELIST_RANDOM=yCONFIG_SLAB_FREELIST_HARDENED=y
# CONFIG_SLUB_STATS is not setCONFIG_SLUB_CPU_PARTIAL=y
# CONFIG_RANDOM_KMALLOC_CACHES is not set
# end of Slab allocator options
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
# CONFIG_COMPAT_BRK is not setCONFIG_SPARSEMEM=yCONFIG_SPARSEMEM_EXTREME=yCONFIG_SPARSEMEM_VMEMMAP_ENABLE=yCONFIG_SPARSEMEM_VMEMMAP=yCONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP=yCONFIG_ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP=yCONFIG_HAVE_FAST_GUP=yCONFIG_NUMA_KEEP_MEMINFO=yCONFIG_MEMORY_ISOLATION=yCONFIG_EXCLUSIVE_SYSTEM_RAM=yCONFIG_HAVE_BOOTMEM_INFO_NODE=yCONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=yCONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=yCONFIG_MEMORY_HOTPLUG=yCONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=yCONFIG_MEMORY_HOTREMOVE=yCONFIG_MHP_MEMMAP_ON_MEMORY=yCONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=yCONFIG_SPLIT_PTLOCK_CPUS=4CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=yCONFIG_MEMORY_BALLOON=yCONFIG_BALLOON_COMPACTION=yCONFIG_COMPACTION=yCONFIG_COMPACT_UNEVICTABLE_DEFAULT=1CONFIG_PAGE_REPORTING=yCONFIG_MIGRATION=yCONFIG_DEVICE_MIGRATION=yCONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=yCONFIG_ARCH_ENABLE_THP_MIGRATION=yCONFIG_CONTIG_ALLOC=yCONFIG_PCP_BATCH_SCALE_MAX=5CONFIG_PHYS_ADDR_T_64BIT=yCONFIG_MMU_NOTIFIER=yCONFIG_KSM=yCONFIG_DEFAULT_MMAP_MIN_ADDR=65536CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=yCONFIG_MEMORY_FAILURE=yCONFIG_HWPOISON_INJECT=mCONFIG_ARCH_WANT_GENERAL_HUGETLB=yCONFIG_ARCH_WANTS_THP_SWAP=yCONFIG_TRANSPARENT_HUGEPAGE=y
# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not setCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
# CONFIG_TRANSPARENT_HUGEPAGE_NEVER is not setCONFIG_THP_SWAP=y
# CONFIG_READ_ONLY_THP_FOR_FS is not setCONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=yCONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=yCONFIG_USE_PERCPU_NUMA_NODE_ID=yCONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_CMA is not setCONFIG_MEM_SOFT_DIRTY=yCONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_DEFERRED_STRUCT_PAGE_INIT is not setCONFIG_PAGE_IDLE_FLAG=yCONFIG_IDLE_PAGE_TRACKING=yCONFIG_ARCH_HAS_CACHE_LINE_SIZE=yCONFIG_ARCH_HAS_CURRENT_STACK_POINTER=yCONFIG_ARCH_HAS_PTE_DEVMAP=yCONFIG_ARCH_HAS_ZONE_DMA_SET=yCONFIG_ZONE_DMA=yCONFIG_ZONE_DMA32=yCONFIG_ZONE_DEVICE=yCONFIG_HMM_MIRROR=yCONFIG_GET_FREE_REGION=yCONFIG_DEVICE_PRIVATE=yCONFIG_VMAP_PFN=yCONFIG_ARCH_USES_HIGH_VMA_FLAGS=yCONFIG_ARCH_HAS_PKEYS=yCONFIG_VM_EVENT_COUNTERS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_TEST is not set
# CONFIG_DMAPOOL_TEST is not setCONFIG_ARCH_HAS_PTE_SPECIAL=yCONFIG_MAPPING_DIRTY_HELPERS=yCONFIG_MEMFD_CREATE=yCONFIG_SECRETMEM=y
# CONFIG_ANON_VMA_NAME is not setCONFIG_HAVE_ARCH_USERFAULTFD_WP=yCONFIG_HAVE_ARCH_USERFAULTFD_MINOR=yCONFIG_USERFAULTFD=yCONFIG_PTE_MARKER_UFFD_WP=y
# CONFIG_LRU_GEN is not setCONFIG_ARCH_SUPPORTS_PER_VMA_LOCK=yCONFIG_PER_VMA_LOCK=yCONFIG_LOCK_MM_AND_FIND_VMA=yCONFIG_IOMMU_MM_DATA=y
#
# Data Access Monitoring
#
# CONFIG_DAMON is not set
# end of Data Access Monitoring # end of Memory Management options
CONFIG_NET=yCONFIG_WANT_COMPAT_NETLINK_MESSAGES=yCONFIG_COMPAT_NETLINK_MESSAGES=yCONFIG_NET_INGRESS=yCONFIG_NET_EGRESS=yCONFIG_NET_XGRESS=yCONFIG_NET_REDIRECT=yCONFIG_SKB_EXTENSIONS=y
#
# Networking options
#CONFIG_PACKET=yCONFIG_PACKET_DIAG=mCONFIG_UNIX=yCONFIG_AF_UNIX_OOB=yCONFIG_UNIX_DIAG=mCONFIG_TLS=mCONFIG_TLS_DEVICE=y
# CONFIG_TLS_TOE is not setCONFIG_XFRM=yCONFIG_XFRM_OFFLOAD=yCONFIG_XFRM_ALGO=mCONFIG_XFRM_USER=mCONFIG_XFRM_USER_COMPAT=mCONFIG_XFRM_INTERFACE=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not setCONFIG_XFRM_STATISTICS=yCONFIG_XFRM_AH=mCONFIG_XFRM_ESP=mCONFIG_XFRM_IPCOMP=mCONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not setCONFIG_XFRM_ESPINTCP=yCONFIG_SMC=mCONFIG_SMC_DIAG=mCONFIG_XDP_SOCKETS=yCONFIG_XDP_SOCKETS_DIAG=mCONFIG_NET_HANDSHAKE=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_FIB_TRIE_STATS=yCONFIG_IP_MULTIPLE_TABLES=yCONFIG_IP_ROUTE_MULTIPATH=yCONFIG_IP_ROUTE_VERBOSE=yCONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not setCONFIG_NET_IPIP=mCONFIG_NET_IPGRE_DEMUX=mCONFIG_NET_IP_TUNNEL=mCONFIG_NET_IPGRE=mCONFIG_NET_IPGRE_BROADCAST=yCONFIG_IP_MROUTE_COMMON=yCONFIG_IP_MROUTE=yCONFIG_IP_MROUTE_MULTIPLE_TABLES=yCONFIG_IP_PIMSM_V1=yCONFIG_IP_PIMSM_V2=yCONFIG_SYN_COOKIES=yCONFIG_NET_IPVTI=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NET_FOU=mCONFIG_NET_FOU_IP_TUNNELS=yCONFIG_INET_AH=mCONFIG_INET_ESP=mCONFIG_INET_ESP_OFFLOAD=mCONFIG_INET_ESPINTCP=yCONFIG_INET_IPCOMP=mCONFIG_INET_TABLE_PERTURB_ORDER=16CONFIG_INET_XFRM_TUNNEL=mCONFIG_INET_TUNNEL=mCONFIG_INET_DIAG=mCONFIG_INET_TCP_DIAG=mCONFIG_INET_UDP_DIAG=mCONFIG_INET_RAW_DIAG=mCONFIG_INET_DIAG_DESTROY=yCONFIG_TCP_CONG_ADVANCED=yCONFIG_TCP_CONG_BIC=mCONFIG_TCP_CONG_CUBIC=yCONFIG_TCP_CONG_WESTWOOD=mCONFIG_TCP_CONG_HTCP=mCONFIG_TCP_CONG_HSTCP=mCONFIG_TCP_CONG_HYBLA=mCONFIG_TCP_CONG_VEGAS=mCONFIG_TCP_CONG_NV=mCONFIG_TCP_CONG_SCALABLE=mCONFIG_TCP_CONG_LP=mCONFIG_TCP_CONG_VENO=mCONFIG_TCP_CONG_YEAH=mCONFIG_TCP_CONG_ILLINOIS=mCONFIG_TCP_CONG_DCTCP=mCONFIG_TCP_CONG_CDG=mCONFIG_TCP_CONG_BBR=mCONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not setCONFIG_DEFAULT_TCP_CONG="cubic"CONFIG_TCP_SIGPOOL=y
# CONFIG_TCP_AO is not setCONFIG_TCP_MD5SIG=yCONFIG_IPV6=yCONFIG_IPV6_ROUTER_PREF=yCONFIG_IPV6_ROUTE_INFO=y
# CONFIG_IPV6_OPTIMISTIC_DAD is not setCONFIG_INET6_AH=mCONFIG_INET6_ESP=mCONFIG_INET6_ESP_OFFLOAD=mCONFIG_INET6_ESPINTCP=yCONFIG_INET6_IPCOMP=mCONFIG_IPV6_MIP6=mCONFIG_IPV6_ILA=mCONFIG_INET6_XFRM_TUNNEL=mCONFIG_INET6_TUNNEL=mCONFIG_IPV6_VTI=mCONFIG_IPV6_SIT=mCONFIG_IPV6_SIT_6RD=yCONFIG_IPV6_NDISC_NODETYPE=yCONFIG_IPV6_TUNNEL=mCONFIG_IPV6_GRE=mCONFIG_IPV6_FOU=mCONFIG_IPV6_FOU_TUNNEL=mCONFIG_IPV6_MULTIPLE_TABLES=yCONFIG_IPV6_SUBTREES=yCONFIG_IPV6_MROUTE=yCONFIG_IPV6_MROUTE_MULTIPLE_TABLES=yCONFIG_IPV6_PIMSM_V2=yCONFIG_IPV6_SEG6_LWTUNNEL=yCONFIG_IPV6_SEG6_HMAC=yCONFIG_IPV6_SEG6_BPF=y
# CONFIG_IPV6_RPL_LWTUNNEL is not setCONFIG_IPV6_IOAM6_LWTUNNEL=yCONFIG_NETLABEL=yCONFIG_MPTCP=yCONFIG_INET_MPTCP_DIAG=mCONFIG_MPTCP_IPV6=yCONFIG_NETWORK_SECMARK=yCONFIG_NET_PTP_CLASSIFY=yCONFIG_NETWORK_PHY_TIMESTAMPING=yCONFIG_NETFILTER=yCONFIG_NETFILTER_ADVANCED=yCONFIG_BRIDGE_NETFILTER=m
#
# Core Netfilter Configuration
#CONFIG_NETFILTER_INGRESS=yCONFIG_NETFILTER_EGRESS=yCONFIG_NETFILTER_SKIP_EGRESS=yCONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_FAMILY_BRIDGE=yCONFIG_NETFILTER_FAMILY_ARP=yCONFIG_NETFILTER_BPF_LINK=yCONFIG_NETFILTER_NETLINK_HOOK=mCONFIG_NETFILTER_NETLINK_ACCT=mCONFIG_NETFILTER_NETLINK_QUEUE=mCONFIG_NETFILTER_NETLINK_LOG=mCONFIG_NETFILTER_NETLINK_OSF=mCONFIG_NF_CONNTRACK=mCONFIG_NF_LOG_SYSLOG=mCONFIG_NETFILTER_CONNCOUNT=mCONFIG_NF_CONNTRACK_MARK=yCONFIG_NF_CONNTRACK_SECMARK=yCONFIG_NF_CONNTRACK_ZONES=y
# CONFIG_NF_CONNTRACK_PROCFS is not setCONFIG_NF_CONNTRACK_EVENTS=yCONFIG_NF_CONNTRACK_TIMEOUT=yCONFIG_NF_CONNTRACK_TIMESTAMP=yCONFIG_NF_CONNTRACK_LABELS=yCONFIG_NF_CONNTRACK_OVS=yCONFIG_NF_CT_PROTO_DCCP=yCONFIG_NF_CT_PROTO_GRE=yCONFIG_NF_CT_PROTO_SCTP=yCONFIG_NF_CT_PROTO_UDPLITE=yCONFIG_NF_CONNTRACK_AMANDA=mCONFIG_NF_CONNTRACK_FTP=mCONFIG_NF_CONNTRACK_H323=mCONFIG_NF_CONNTRACK_IRC=mCONFIG_NF_CONNTRACK_BROADCAST=mCONFIG_NF_CONNTRACK_NETBIOS_NS=mCONFIG_NF_CONNTRACK_SNMP=mCONFIG_NF_CONNTRACK_PPTP=mCONFIG_NF_CONNTRACK_SANE=mCONFIG_NF_CONNTRACK_SIP=mCONFIG_NF_CONNTRACK_TFTP=mCONFIG_NF_CT_NETLINK=mCONFIG_NF_CT_NETLINK_TIMEOUT=mCONFIG_NF_CT_NETLINK_HELPER=mCONFIG_NETFILTER_NETLINK_GLUE_CT=yCONFIG_NF_NAT=mCONFIG_NF_NAT_AMANDA=mCONFIG_NF_NAT_FTP=mCONFIG_NF_NAT_IRC=mCONFIG_NF_NAT_SIP=mCONFIG_NF_NAT_TFTP=mCONFIG_NF_NAT_REDIRECT=yCONFIG_NF_NAT_MASQUERADE=yCONFIG_NF_NAT_OVS=yCONFIG_NETFILTER_SYNPROXY=mCONFIG_NF_TABLES=mCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_NUMGEN=mCONFIG_NFT_CT=mCONFIG_NFT_FLOW_OFFLOAD=mCONFIG_NFT_CONNLIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_LIMIT=mCONFIG_NFT_MASQ=mCONFIG_NFT_REDIR=mCONFIG_NFT_NAT=mCONFIG_NFT_TUNNEL=mCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REJECT=mCONFIG_NFT_REJECT_INET=mCONFIG_NFT_COMPAT=mCONFIG_NFT_HASH=mCONFIG_NFT_FIB=mCONFIG_NFT_FIB_INET=mCONFIG_NFT_XFRM=mCONFIG_NFT_SOCKET=mCONFIG_NFT_OSF=mCONFIG_NFT_TPROXY=mCONFIG_NFT_SYNPROXY=mCONFIG_NF_DUP_NETDEV=mCONFIG_NFT_DUP_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_REJECT_NETDEV=mCONFIG_NF_FLOW_TABLE_INET=mCONFIG_NF_FLOW_TABLE=m
# CONFIG_NF_FLOW_TABLE_PROCFS is not setCONFIG_NETFILTER_XTABLES=mCONFIG_NETFILTER_XTABLES_COMPAT=y
#
# Xtables combined modules
#CONFIG_NETFILTER_XT_MARK=mCONFIG_NETFILTER_XT_CONNMARK=mCONFIG_NETFILTER_XT_SET=m
#
# Xtables targets
#CONFIG_NETFILTER_XT_TARGET_AUDIT=mCONFIG_NETFILTER_XT_TARGET_CHECKSUM=mCONFIG_NETFILTER_XT_TARGET_CLASSIFY=mCONFIG_NETFILTER_XT_TARGET_CONNMARK=mCONFIG_NETFILTER_XT_TARGET_CONNSECMARK=mCONFIG_NETFILTER_XT_TARGET_CT=mCONFIG_NETFILTER_XT_TARGET_DSCP=mCONFIG_NETFILTER_XT_TARGET_HL=mCONFIG_NETFILTER_XT_TARGET_HMARK=mCONFIG_NETFILTER_XT_TARGET_IDLETIMER=mCONFIG_NETFILTER_XT_TARGET_LED=mCONFIG_NETFILTER_XT_TARGET_LOG=mCONFIG_NETFILTER_XT_TARGET_MARK=mCONFIG_NETFILTER_XT_NAT=mCONFIG_NETFILTER_XT_TARGET_NETMAP=mCONFIG_NETFILTER_XT_TARGET_NFLOG=mCONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
# CONFIG_NETFILTER_XT_TARGET_NOTRACK is not setCONFIG_NETFILTER_XT_TARGET_RATEEST=mCONFIG_NETFILTER_XT_TARGET_REDIRECT=mCONFIG_NETFILTER_XT_TARGET_MASQUERADE=mCONFIG_NETFILTER_XT_TARGET_TEE=mCONFIG_NETFILTER_XT_TARGET_TPROXY=mCONFIG_NETFILTER_XT_TARGET_TRACE=mCONFIG_NETFILTER_XT_TARGET_SECMARK=mCONFIG_NETFILTER_XT_TARGET_TCPMSS=mCONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
#
# Xtables matches
#CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=mCONFIG_NETFILTER_XT_MATCH_BPF=mCONFIG_NETFILTER_XT_MATCH_CGROUP=mCONFIG_NETFILTER_XT_MATCH_CLUSTER=mCONFIG_NETFILTER_XT_MATCH_COMMENT=mCONFIG_NETFILTER_XT_MATCH_CONNBYTES=mCONFIG_NETFILTER_XT_MATCH_CONNLABEL=mCONFIG_NETFILTER_XT_MATCH_CONNLIMIT=mCONFIG_NETFILTER_XT_MATCH_CONNMARK=mCONFIG_NETFILTER_XT_MATCH_CONNTRACK=mCONFIG_NETFILTER_XT_MATCH_CPU=mCONFIG_NETFILTER_XT_MATCH_DCCP=mCONFIG_NETFILTER_XT_MATCH_DEVGROUP=mCONFIG_NETFILTER_XT_MATCH_DSCP=mCONFIG_NETFILTER_XT_MATCH_ECN=mCONFIG_NETFILTER_XT_MATCH_ESP=mCONFIG_NETFILTER_XT_MATCH_HASHLIMIT=mCONFIG_NETFILTER_XT_MATCH_HELPER=mCONFIG_NETFILTER_XT_MATCH_HL=mCONFIG_NETFILTER_XT_MATCH_IPCOMP=mCONFIG_NETFILTER_XT_MATCH_IPRANGE=mCONFIG_NETFILTER_XT_MATCH_IPVS=mCONFIG_NETFILTER_XT_MATCH_L2TP=mCONFIG_NETFILTER_XT_MATCH_LENGTH=mCONFIG_NETFILTER_XT_MATCH_LIMIT=mCONFIG_NETFILTER_XT_MATCH_MAC=mCONFIG_NETFILTER_XT_MATCH_MARK=mCONFIG_NETFILTER_XT_MATCH_MULTIPORT=mCONFIG_NETFILTER_XT_MATCH_NFACCT=mCONFIG_NETFILTER_XT_MATCH_OSF=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NETFILTER_XT_MATCH_POLICY=mCONFIG_NETFILTER_XT_MATCH_PHYSDEV=mCONFIG_NETFILTER_XT_MATCH_PKTTYPE=mCONFIG_NETFILTER_XT_MATCH_QUOTA=mCONFIG_NETFILTER_XT_MATCH_RATEEST=mCONFIG_NETFILTER_XT_MATCH_REALM=mCONFIG_NETFILTER_XT_MATCH_RECENT=mCONFIG_NETFILTER_XT_MATCH_SCTP=mCONFIG_NETFILTER_XT_MATCH_SOCKET=mCONFIG_NETFILTER_XT_MATCH_STATE=mCONFIG_NETFILTER_XT_MATCH_STATISTIC=mCONFIG_NETFILTER_XT_MATCH_STRING=mCONFIG_NETFILTER_XT_MATCH_TCPMSS=mCONFIG_NETFILTER_XT_MATCH_TIME=mCONFIG_NETFILTER_XT_MATCH_U32=m
# end of Core Netfilter Configuration
CONFIG_IP_SET=mCONFIG_IP_SET_MAX=256CONFIG_IP_SET_BITMAP_IP=mCONFIG_IP_SET_BITMAP_IPMAC=mCONFIG_IP_SET_BITMAP_PORT=mCONFIG_IP_SET_HASH_IP=mCONFIG_IP_SET_HASH_IPMARK=mCONFIG_IP_SET_HASH_IPPORT=mCONFIG_IP_SET_HASH_IPPORTIP=mCONFIG_IP_SET_HASH_IPPORTNET=mCONFIG_IP_SET_HASH_IPMAC=mCONFIG_IP_SET_HASH_MAC=mCONFIG_IP_SET_HASH_NETPORTNET=mCONFIG_IP_SET_HASH_NET=mCONFIG_IP_SET_HASH_NETNET=mCONFIG_IP_SET_HASH_NETPORT=mCONFIG_IP_SET_HASH_NETIFACE=mCONFIG_IP_SET_LIST_SET=mCONFIG_IP_VS=mCONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not setCONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#CONFIG_IP_VS_PROTO_TCP=yCONFIG_IP_VS_PROTO_UDP=yCONFIG_IP_VS_PROTO_AH_ESP=yCONFIG_IP_VS_PROTO_ESP=yCONFIG_IP_VS_PROTO_AH=yCONFIG_IP_VS_PROTO_SCTP=y
#
# IPVS scheduler
#CONFIG_IP_VS_RR=mCONFIG_IP_VS_WRR=mCONFIG_IP_VS_LC=mCONFIG_IP_VS_WLC=mCONFIG_IP_VS_FO=mCONFIG_IP_VS_OVF=mCONFIG_IP_VS_LBLC=mCONFIG_IP_VS_LBLCR=mCONFIG_IP_VS_DH=mCONFIG_IP_VS_SH=mCONFIG_IP_VS_MH=mCONFIG_IP_VS_SED=mCONFIG_IP_VS_NQ=mCONFIG_IP_VS_TWOS=m
#
# IPVS SH scheduler
#CONFIG_IP_VS_SH_TAB_BITS=8
#
# IPVS MH scheduler
#CONFIG_IP_VS_MH_TAB_INDEX=12
#
# IPVS application helper
#CONFIG_IP_VS_FTP=mCONFIG_IP_VS_NFCT=yCONFIG_IP_VS_PE_SIP=m
#
# IP: Netfilter Configuration
#CONFIG_NF_DEFRAG_IPV4=mCONFIG_IP_NF_IPTABLES_LEGACY=mCONFIG_NF_SOCKET_IPV4=mCONFIG_NF_TPROXY_IPV4=mCONFIG_NF_TABLES_IPV4=yCONFIG_NFT_REJECT_IPV4=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_FIB_IPV4=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_DUP_IPV4=mCONFIG_NF_LOG_ARP=mCONFIG_NF_LOG_IPV4=mCONFIG_NF_REJECT_IPV4=mCONFIG_NF_NAT_SNMP_BASIC=mCONFIG_NF_NAT_PPTP=mCONFIG_NF_NAT_H323=mCONFIG_IP_NF_IPTABLES=mCONFIG_IP_NF_MATCH_AH=mCONFIG_IP_NF_MATCH_ECN=mCONFIG_IP_NF_MATCH_RPFILTER=mCONFIG_IP_NF_MATCH_TTL=mCONFIG_IP_NF_FILTER=mCONFIG_IP_NF_TARGET_REJECT=mCONFIG_IP_NF_TARGET_SYNPROXY=mCONFIG_IP_NF_NAT=mCONFIG_IP_NF_TARGET_MASQUERADE=mCONFIG_IP_NF_TARGET_NETMAP=mCONFIG_IP_NF_TARGET_REDIRECT=mCONFIG_IP_NF_MANGLE=mCONFIG_IP_NF_TARGET_ECN=mCONFIG_IP_NF_TARGET_TTL=mCONFIG_IP_NF_RAW=mCONFIG_IP_NF_SECURITY=mCONFIG_IP_NF_ARPTABLES=mCONFIG_NFT_COMPAT_ARP=mCONFIG_IP_NF_ARPFILTER=mCONFIG_IP_NF_ARP_MANGLE=m
# end of IP: Netfilter Configuration
#
# IPv6: Netfilter Configuration
#CONFIG_IP6_NF_IPTABLES_LEGACY=mCONFIG_NF_SOCKET_IPV6=mCONFIG_NF_TPROXY_IPV6=mCONFIG_NF_TABLES_IPV6=yCONFIG_NFT_REJECT_IPV6=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_FIB_IPV6=mCONFIG_NF_DUP_IPV6=mCONFIG_NF_REJECT_IPV6=mCONFIG_NF_LOG_IPV6=mCONFIG_IP6_NF_IPTABLES=mCONFIG_IP6_NF_MATCH_AH=mCONFIG_IP6_NF_MATCH_EUI64=mCONFIG_IP6_NF_MATCH_FRAG=mCONFIG_IP6_NF_MATCH_OPTS=mCONFIG_IP6_NF_MATCH_HL=mCONFIG_IP6_NF_MATCH_IPV6HEADER=mCONFIG_IP6_NF_MATCH_MH=mCONFIG_IP6_NF_MATCH_RPFILTER=mCONFIG_IP6_NF_MATCH_RT=mCONFIG_IP6_NF_MATCH_SRH=mCONFIG_IP6_NF_TARGET_HL=mCONFIG_IP6_NF_FILTER=mCONFIG_IP6_NF_TARGET_REJECT=mCONFIG_IP6_NF_TARGET_SYNPROXY=mCONFIG_IP6_NF_MANGLE=mCONFIG_IP6_NF_RAW=mCONFIG_IP6_NF_SECURITY=mCONFIG_IP6_NF_NAT=mCONFIG_IP6_NF_TARGET_MASQUERADE=mCONFIG_IP6_NF_TARGET_NPT=m
# end of IPv6: Netfilter Configuration
CONFIG_NF_DEFRAG_IPV6=mCONFIG_NF_TABLES_BRIDGE=mCONFIG_NFT_BRIDGE_META=mCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NF_CONNTRACK_BRIDGE=mCONFIG_BRIDGE_NF_EBTABLES_LEGACY=mCONFIG_BRIDGE_NF_EBTABLES=mCONFIG_BRIDGE_EBT_BROUTE=mCONFIG_BRIDGE_EBT_T_FILTER=mCONFIG_BRIDGE_EBT_T_NAT=mCONFIG_BRIDGE_EBT_802_3=mCONFIG_BRIDGE_EBT_AMONG=mCONFIG_BRIDGE_EBT_ARP=mCONFIG_BRIDGE_EBT_IP=mCONFIG_BRIDGE_EBT_IP6=mCONFIG_BRIDGE_EBT_LIMIT=mCONFIG_BRIDGE_EBT_MARK=mCONFIG_BRIDGE_EBT_PKTTYPE=mCONFIG_BRIDGE_EBT_STP=mCONFIG_BRIDGE_EBT_VLAN=mCONFIG_BRIDGE_EBT_ARPREPLY=mCONFIG_BRIDGE_EBT_DNAT=mCONFIG_BRIDGE_EBT_MARK_T=mCONFIG_BRIDGE_EBT_REDIRECT=mCONFIG_BRIDGE_EBT_SNAT=mCONFIG_BRIDGE_EBT_LOG=mCONFIG_BRIDGE_EBT_NFLOG=mCONFIG_IP_DCCP=mCONFIG_INET_DCCP_DIAG=m
#
# DCCP CCIDs Configuration
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
# CONFIG_IP_DCCP_CCID3 is not set
# end of DCCP CCIDs Configuration
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# end of DCCP Kernel Hacking
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5 is not set CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set CONFIG_SCTP_COOKIE_HMAC_MD5=y CONFIG_SCTP_COOKIE_HMAC_SHA1=y CONFIG_INET_SCTP_DIAG=mCONFIG_RDS=mCONFIG_RDS_RDMA=mCONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not setCONFIG_TIPC=mCONFIG_TIPC_MEDIA_IB=y CONFIG_TIPC_MEDIA_UDP=yCONFIG_TIPC_CRYPTO=yCONFIG_TIPC_DIAG=mCONFIG_ATM=mCONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set CONFIG_ATM_LANE=mCONFIG_ATM_MPOA=mCONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set CONFIG_L2TP=mCONFIG_L2TP_DEBUGFS=mCONFIG_L2TP_V3=yCONFIG_L2TP_IP=mCONFIG_L2TP_ETH=mCONFIG_STP=mCONFIG_GARP=mCONFIG_MRP=mCONFIG_BRIDGE=m CONFIG_BRIDGE_IGMP_SNOOPING=y CONFIG_BRIDGE_VLAN_FILTERING=y CONFIG_BRIDGE_MRP=yCONFIG_BRIDGE_CFM=yCONFIG_NET_DSA=mCONFIG_NET_DSA_TAG_NONE=m CONFIG_NET_DSA_TAG_AR9331=m CONFIG_NET_DSA_TAG_BRCM_COMMON=m CONFIG_NET_DSA_TAG_BRCM=m CONFIG_NET_DSA_TAG_BRCM_LEGACY=m CONFIG_NET_DSA_TAG_BRCM_PREPEND=m CONFIG_NET_DSA_TAG_HELLCREEK=m CONFIG_NET_DSA_TAG_GSWIP=m CONFIG_NET_DSA_TAG_DSA_COMMON=m CONFIG_NET_DSA_TAG_DSA=m CONFIG_NET_DSA_TAG_EDSA=m CONFIG_NET_DSA_TAG_MTK=m CONFIG_NET_DSA_TAG_KSZ=m CONFIG_NET_DSA_TAG_OCELOT=m CONFIG_NET_DSA_TAG_OCELOT_8021Q=m CONFIG_NET_DSA_TAG_QCA=m CONFIG_NET_DSA_TAG_RTL4_A=m
# CONFIG_NET_DSA_TAG_RTL8_4 is not set
# CONFIG_NET_DSA_TAG_RZN1_A5PSW is not set CONFIG_NET_DSA_TAG_LAN9303=m CONFIG_NET_DSA_TAG_SJA1105=m CONFIG_NET_DSA_TAG_TRAILER=m CONFIG_NET_DSA_TAG_XRS700X=m CONFIG_VLAN_8021Q=mCONFIG_VLAN_8021Q_GVRP=y CONFIG_VLAN_8021Q_MVRP=yCONFIG_LLC=mCONFIG_LLC2=mCONFIG_ATALK=mCONFIG_X25=mCONFIG_LAPB=mCONFIG_PHONET=mCONFIG_6LOWPAN=m
# CONFIG_6LOWPAN_DEBUGFS is not set CONFIG_6LOWPAN_NHC=m CONFIG_6LOWPAN_NHC_DEST=m CONFIG_6LOWPAN_NHC_FRAGMENT=m CONFIG_6LOWPAN_NHC_HOP=m CONFIG_6LOWPAN_NHC_IPV6=m CONFIG_6LOWPAN_NHC_MOBILITY=m CONFIG_6LOWPAN_NHC_ROUTING=m CONFIG_6LOWPAN_NHC_UDP=m
# CONFIG_6LOWPAN_GHC_EXT_HDR_HOP is not set # CONFIG_6LOWPAN_GHC_UDP is not set
# CONFIG_6LOWPAN_GHC_ICMPV6 is not set
# CONFIG_6LOWPAN_GHC_EXT_HDR_DEST is not set # CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG is not set # CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE is not set CONFIG_IEEE802154=m
# CONFIG_IEEE802154_NL802154_EXPERIMENTAL is not set CONFIG_IEEE802154_SOCKET=m CONFIG_IEEE802154_6LOWPAN=m CONFIG_MAC802154=mCONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#CONFIG_NET_SCH_HTB=mCONFIG_NET_SCH_HFSC=mCONFIG_NET_SCH_PRIO=mCONFIG_NET_SCH_MULTIQ=mCONFIG_NET_SCH_RED=mCONFIG_NET_SCH_SFB=mCONFIG_NET_SCH_SFQ=mCONFIG_NET_SCH_TEQL=mCONFIG_NET_SCH_TBF=mCONFIG_NET_SCH_CBS=mCONFIG_NET_SCH_ETF=mCONFIG_NET_SCH_MQPRIO_LIB=mCONFIG_NET_SCH_TAPRIO=mCONFIG_NET_SCH_GRED=mCONFIG_NET_SCH_NETEM=mCONFIG_NET_SCH_DRR=mCONFIG_NET_SCH_MQPRIO=mCONFIG_NET_SCH_SKBPRIO=mCONFIG_NET_SCH_CHOKE=mCONFIG_NET_SCH_QFQ=mCONFIG_NET_SCH_CODEL=mCONFIG_NET_SCH_FQ_CODEL=mCONFIG_NET_SCH_CAKE=mCONFIG_NET_SCH_FQ=mCONFIG_NET_SCH_HHF=mCONFIG_NET_SCH_PIE=mCONFIG_NET_SCH_FQ_PIE=mCONFIG_NET_SCH_INGRESS=mCONFIG_NET_SCH_PLUG=mCONFIG_NET_SCH_ETS=m
# CONFIG_NET_SCH_DEFAULT is not set
#
# Classification
#CONFIG_NET_CLS=yCONFIG_NET_CLS_BASIC=mCONFIG_NET_CLS_ROUTE4=mCONFIG_NET_CLS_FW=mCONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not setCONFIG_CLS_U32_MARK=yCONFIG_NET_CLS_FLOW=mCONFIG_NET_CLS_CGROUP=mCONFIG_NET_CLS_BPF=mCONFIG_NET_CLS_FLOWER=mCONFIG_NET_CLS_MATCHALL=mCONFIG_NET_EMATCH=yCONFIG_NET_EMATCH_STACK=32CONFIG_NET_EMATCH_CMP=mCONFIG_NET_EMATCH_NBYTE=mCONFIG_NET_EMATCH_U32=mCONFIG_NET_EMATCH_META=mCONFIG_NET_EMATCH_TEXT=mCONFIG_NET_EMATCH_CANID=mCONFIG_NET_EMATCH_IPSET=mCONFIG_NET_EMATCH_IPT=mCONFIG_NET_CLS_ACT=yCONFIG_NET_ACT_POLICE=mCONFIG_NET_ACT_GACT=mCONFIG_GACT_PROB=yCONFIG_NET_ACT_MIRRED=mCONFIG_NET_ACT_SAMPLE=mCONFIG_NET_ACT_NAT=mCONFIG_NET_ACT_PEDIT=mCONFIG_NET_ACT_SIMP=mCONFIG_NET_ACT_SKBEDIT=mCONFIG_NET_ACT_CSUM=mCONFIG_NET_ACT_MPLS=mCONFIG_NET_ACT_VLAN=mCONFIG_NET_ACT_BPF=mCONFIG_NET_ACT_CONNMARK=mCONFIG_NET_ACT_CTINFO=mCONFIG_NET_ACT_SKBMOD=m
# CONFIG_NET_ACT_IFE is not setCONFIG_NET_ACT_TUNNEL_KEY=mCONFIG_NET_ACT_CT=mCONFIG_NET_ACT_GATE=mCONFIG_NET_TC_SKB_EXT=yCONFIG_NET_SCH_FIFO=yCONFIG_DCB=yCONFIG_DNS_RESOLVER=yCONFIG_BATMAN_ADV=m
# CONFIG_BATMAN_ADV_BATMAN_V is not setCONFIG_BATMAN_ADV_BLA=yCONFIG_BATMAN_ADV_DAT=yCONFIG_BATMAN_ADV_NC=yCONFIG_BATMAN_ADV_MCAST=y
# CONFIG_BATMAN_ADV_DEBUG is not set
# CONFIG_BATMAN_ADV_TRACING is not setCONFIG_OPENVSWITCH=mCONFIG_OPENVSWITCH_GRE=mCONFIG_OPENVSWITCH_VXLAN=mCONFIG_OPENVSWITCH_GENEVE=mCONFIG_VSOCKETS=mCONFIG_VSOCKETS_DIAG=mCONFIG_VSOCKETS_LOOPBACK=mCONFIG_VMWARE_VMCI_VSOCKETS=mCONFIG_VIRTIO_VSOCKETS=mCONFIG_VIRTIO_VSOCKETS_COMMON=mCONFIG_HYPERV_VSOCKETS=mCONFIG_NETLINK_DIAG=mCONFIG_MPLS=yCONFIG_NET_MPLS_GSO=mCONFIG_MPLS_ROUTING=mCONFIG_MPLS_IPTUNNEL=mCONFIG_NET_NSH=mCONFIG_HSR=mCONFIG_NET_SWITCHDEV=yCONFIG_NET_L3_MASTER_DEV=yCONFIG_QRTR=mCONFIG_QRTR_SMD=mCONFIG_QRTR_TUN=mCONFIG_QRTR_MHI=mCONFIG_NET_NCSI=yCONFIG_NCSI_OEM_CMD_GET_MAC=y
# CONFIG_NCSI_OEM_CMD_KEEP_PHY is not setCONFIG_PCPU_DEV_REFCNT=yCONFIG_MAX_SKB_FRAGS=17CONFIG_RPS=yCONFIG_RFS_ACCEL=yCONFIG_SOCK_RX_QUEUE_MAPPING=yCONFIG_XPS=yCONFIG_CGROUP_NET_PRIO=yCONFIG_CGROUP_NET_CLASSID=yCONFIG_NET_RX_BUSY_POLL=yCONFIG_BQL=yCONFIG_BPF_STREAM_PARSER=yCONFIG_NET_FLOW_LIMIT=y
#
# Network testing
#CONFIG_NET_PKTGEN=mCONFIG_NET_DROP_MONITOR=y
# end of Network testing # end of Networking options
CONFIG_HAMRADIO=y
#
# Packet Radio protocols
#CONFIG_AX25=mCONFIG_AX25_DAMA_SLAVE=yCONFIG_NETROM=mCONFIG_ROSE=m
#
# AX.25 network device drivers
#CONFIG_MKISS=mCONFIG_6PACK=mCONFIG_BPQETHER=mCONFIG_BAYCOM_SER_FDX=mCONFIG_BAYCOM_SER_HDX=mCONFIG_BAYCOM_PAR=mCONFIG_YAM=m
# end of AX.25 network device drivers
CONFIG_CAN=mCONFIG_CAN_RAW=mCONFIG_CAN_BCM=mCONFIG_CAN_GW=mCONFIG_CAN_J1939=mCONFIG_CAN_ISOTP=mCONFIG_BT=mCONFIG_BT_BREDR=yCONFIG_BT_RFCOMM=mCONFIG_BT_RFCOMM_TTY=yCONFIG_BT_BNEP=mCONFIG_BT_BNEP_MC_FILTER=yCONFIG_BT_BNEP_PROTO_FILTER=yCONFIG_BT_CMTP=mCONFIG_BT_HIDP=mCONFIG_BT_LE=yCONFIG_BT_LE_L2CAP_ECRED=yCONFIG_BT_6LOWPAN=mCONFIG_BT_LEDS=yCONFIG_BT_MSFTEXT=yCONFIG_BT_AOSPEXT=yCONFIG_BT_DEBUGFS=y
# CONFIG_BT_SELFTEST is not set
#
# Bluetooth device drivers
#CONFIG_BT_INTEL=mCONFIG_BT_BCM=mCONFIG_BT_RTL=mCONFIG_BT_QCA=mCONFIG_BT_MTK=mCONFIG_BT_HCIBTUSB=mCONFIG_BT_HCIBTUSB_AUTOSUSPEND=yCONFIG_BT_HCIBTUSB_POLL_SYNC=yCONFIG_BT_HCIBTUSB_BCM=yCONFIG_BT_HCIBTUSB_MTK=yCONFIG_BT_HCIBTUSB_RTL=yCONFIG_BT_HCIBTSDIO=mCONFIG_BT_HCIUART=mCONFIG_BT_HCIUART_SERDEV=yCONFIG_BT_HCIUART_H4=yCONFIG_BT_HCIUART_NOKIA=mCONFIG_BT_HCIUART_BCSP=yCONFIG_BT_HCIUART_ATH3K=yCONFIG_BT_HCIUART_LL=yCONFIG_BT_HCIUART_3WIRE=yCONFIG_BT_HCIUART_INTEL=yCONFIG_BT_HCIUART_BCM=yCONFIG_BT_HCIUART_RTL=yCONFIG_BT_HCIUART_QCA=yCONFIG_BT_HCIUART_AG6XX=yCONFIG_BT_HCIUART_MRVL=yCONFIG_BT_HCIBCM203X=m
# CONFIG_BT_HCIBCM4377 is not setCONFIG_BT_HCIBPA10X=mCONFIG_BT_HCIBFUSB=mCONFIG_BT_HCIDTL1=mCONFIG_BT_HCIBT3C=mCONFIG_BT_HCIBLUECARD=mCONFIG_BT_HCIVHCI=mCONFIG_BT_MRVL=mCONFIG_BT_MRVL_SDIO=mCONFIG_BT_ATH3K=mCONFIG_BT_MTKSDIO=mCONFIG_BT_MTKUART=mCONFIG_BT_HCIRSI=mCONFIG_BT_VIRTIO=m
# CONFIG_BT_NXPUART is not set
# end of Bluetooth device drivers
CONFIG_AF_RXRPC=mCONFIG_AF_RXRPC_IPV6=y
# CONFIG_AF_RXRPC_INJECT_LOSS is not set
# CONFIG_AF_RXRPC_INJECT_RX_DELAY is not set
# CONFIG_AF_RXRPC_DEBUG is not setCONFIG_RXKAD=y
# CONFIG_RXPERF is not setCONFIG_AF_KCM=mCONFIG_STREAM_PARSER=y
# CONFIG_MCTP is not setCONFIG_FIB_RULES=yCONFIG_WIRELESS=yCONFIG_WIRELESS_EXT=yCONFIG_WEXT_CORE=yCONFIG_WEXT_PROC=yCONFIG_WEXT_SPY=yCONFIG_WEXT_PRIV=yCONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_CERTIFICATION_ONUS is not setCONFIG_CFG80211_REQUIRE_SIGNED_REGDB=yCONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=yCONFIG_CFG80211_DEFAULT_PS=yCONFIG_CFG80211_DEBUGFS=yCONFIG_CFG80211_CRDA_SUPPORT=yCONFIG_CFG80211_WEXT=yCONFIG_CFG80211_WEXT_EXPORT=yCONFIG_LIB80211=mCONFIG_LIB80211_CRYPT_WEP=mCONFIG_LIB80211_CRYPT_CCMP=mCONFIG_LIB80211_CRYPT_TKIP=m
# CONFIG_LIB80211_DEBUG is not setCONFIG_MAC80211=mCONFIG_MAC80211_HAS_RC=yCONFIG_MAC80211_RC_MINSTREL=yCONFIG_MAC80211_RC_DEFAULT_MINSTREL=yCONFIG_MAC80211_RC_DEFAULT="minstrel_ht"CONFIG_MAC80211_MESH=yCONFIG_MAC80211_LEDS=yCONFIG_MAC80211_DEBUGFS=yCONFIG_MAC80211_MESSAGE_TRACING=y
# CONFIG_MAC80211_DEBUG_MENU is not setCONFIG_MAC80211_STA_HASH_MAX_SIZE=0CONFIG_RFKILL=yCONFIG_RFKILL_LEDS=yCONFIG_RFKILL_INPUT=yCONFIG_RFKILL_GPIO=mCONFIG_NET_9P=mCONFIG_NET_9P_FD=mCONFIG_NET_9P_VIRTIO=mCONFIG_NET_9P_XEN=mCONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not setCONFIG_CAIF=m
# CONFIG_CAIF_DEBUG is not setCONFIG_CAIF_NETDEV=mCONFIG_CAIF_USB=mCONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not setCONFIG_CEPH_LIB_USE_DNS_RESOLVER=yCONFIG_NFC=mCONFIG_NFC_DIGITAL=mCONFIG_NFC_NCI=mCONFIG_NFC_NCI_SPI=mCONFIG_NFC_NCI_UART=mCONFIG_NFC_HCI=mCONFIG_NFC_SHDLC=y
#
# Near Field Communication (NFC) devices
#CONFIG_NFC_TRF7970A=mCONFIG_NFC_MEI_PHY=mCONFIG_NFC_SIM=mCONFIG_NFC_PORT100=mCONFIG_NFC_VIRTUAL_NCI=mCONFIG_NFC_FDP=mCONFIG_NFC_FDP_I2C=mCONFIG_NFC_PN544=mCONFIG_NFC_PN544_I2C=mCONFIG_NFC_PN544_MEI=mCONFIG_NFC_PN533=mCONFIG_NFC_PN533_USB=mCONFIG_NFC_PN533_I2C=mCONFIG_NFC_PN532_UART=mCONFIG_NFC_MICROREAD=mCONFIG_NFC_MICROREAD_I2C=mCONFIG_NFC_MICROREAD_MEI=mCONFIG_NFC_MRVL=mCONFIG_NFC_MRVL_USB=mCONFIG_NFC_MRVL_UART=mCONFIG_NFC_MRVL_I2C=mCONFIG_NFC_MRVL_SPI=mCONFIG_NFC_ST21NFCA=mCONFIG_NFC_ST21NFCA_I2C=mCONFIG_NFC_ST_NCI=mCONFIG_NFC_ST_NCI_I2C=mCONFIG_NFC_ST_NCI_SPI=mCONFIG_NFC_NXP_NCI=mCONFIG_NFC_NXP_NCI_I2C=mCONFIG_NFC_S3FWRN5=mCONFIG_NFC_S3FWRN5_I2C=mCONFIG_NFC_S3FWRN82_UART=mCONFIG_NFC_ST95HF=m
# end of Near Field Communication (NFC) devices
CONFIG_PSAMPLE=mCONFIG_NET_IFE=mCONFIG_LWTUNNEL=yCONFIG_LWTUNNEL_BPF=yCONFIG_DST_CACHE=yCONFIG_GRO_CELLS=yCONFIG_SOCK_VALIDATE_XMIT=yCONFIG_NET_SELFTESTS=yCONFIG_NET_SOCK_MSG=yCONFIG_NET_DEVLINK=yCONFIG_PAGE_POOL=y
# CONFIG_PAGE_POOL_STATS is not setCONFIG_FAILOVER=mCONFIG_ETHTOOL_NETLINK=y
#
# Device Drivers
#CONFIG_HAVE_EISA=yCONFIG_EISA=yCONFIG_EISA_VLB_PRIMING=yCONFIG_EISA_PCI_EISA=yCONFIG_EISA_VIRTUAL_ROOT=yCONFIG_EISA_NAMES=yCONFIG_HAVE_PCI=yCONFIG_GENERIC_PCI_IOMAP=yCONFIG_PCI=yCONFIG_PCI_DOMAINS=yCONFIG_PCIEPORTBUS=yCONFIG_HOTPLUG_PCI_PCIE=yCONFIG_PCIEAER=y
# CONFIG_PCIEAER_INJECT is not setCONFIG_PCIEAER_CXL=y
# CONFIG_PCIE_ECRC is not setCONFIG_PCIEASPM=yCONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not setCONFIG_PCIE_PME=yCONFIG_PCIE_DPC=yCONFIG_PCIE_PTM=yCONFIG_PCIE_EDR=yCONFIG_PCI_MSI=yCONFIG_PCI_QUIRKS=y
# CONFIG_PCI_DEBUG is not setCONFIG_PCI_REALLOC_ENABLE_AUTO=yCONFIG_PCI_STUB=mCONFIG_PCI_PF_STUB=mCONFIG_XEN_PCIDEV_FRONTEND=mCONFIG_PCI_ATS=yCONFIG_PCI_DOE=yCONFIG_PCI_LOCKLESS_CONFIG=yCONFIG_PCI_IOV=yCONFIG_PCI_PRI=yCONFIG_PCI_PASID=y
# CONFIG_PCI_P2PDMA is not setCONFIG_PCI_LABEL=yCONFIG_PCI_HYPERV=m
# CONFIG_PCIE_BUS_TUNE_OFF is not setCONFIG_PCIE_BUS_DEFAULT=y
# CONFIG_PCIE_BUS_SAFE is not set
# CONFIG_PCIE_BUS_PERFORMANCE is not set
# CONFIG_PCIE_BUS_PEER2PEER is not setCONFIG_VGA_ARB=yCONFIG_VGA_ARB_MAX_GPUS=16CONFIG_HOTPLUG_PCI=yCONFIG_HOTPLUG_PCI_ACPI=yCONFIG_HOTPLUG_PCI_ACPI_IBM=mCONFIG_HOTPLUG_PCI_CPCI=yCONFIG_HOTPLUG_PCI_CPCI_ZT5550=mCONFIG_HOTPLUG_PCI_CPCI_GENERIC=mCONFIG_HOTPLUG_PCI_SHPC=y
#
# PCI controller drivers
#CONFIG_VMD=mCONFIG_PCI_HYPERV_INTERFACE=m
#
# Cadence-based PCIe controllers
#
# end of Cadence-based PCIe controllers
#
# DesignWare-based PCIe controllers
#CONFIG_PCIE_DW=yCONFIG_PCIE_DW_HOST=yCONFIG_PCIE_DW_EP=y
# CONFIG_PCI_MESON is not setCONFIG_PCIE_DW_PLAT=yCONFIG_PCIE_DW_PLAT_HOST=yCONFIG_PCIE_DW_PLAT_EP=y
# end of DesignWare-based PCIe controllers
#
# Mobiveil-based PCIe controllers
#
# end of Mobiveil-based PCIe controllers # end of PCI controller drivers
#
# PCI Endpoint
#CONFIG_PCI_ENDPOINT=yCONFIG_PCI_ENDPOINT_CONFIGFS=y
# CONFIG_PCI_EPF_TEST is not setCONFIG_PCI_EPF_NTB=m
# CONFIG_PCI_EPF_VNTB is not set
# end of PCI Endpoint
#
# PCI switch controller drivers
#CONFIG_PCI_SW_SWITCHTEC=m
# end of PCI switch controller drivers
CONFIG_CXL_BUS=mCONFIG_CXL_PCI=mCONFIG_CXL_MEM_RAW_COMMANDS=yCONFIG_CXL_ACPI=mCONFIG_CXL_PMEM=mCONFIG_CXL_MEM=mCONFIG_CXL_PORT=mCONFIG_CXL_SUSPEND=yCONFIG_CXL_REGION=y
# CONFIG_CXL_REGION_INVALIDATION_TEST is not setCONFIG_CXL_PMU=mCONFIG_PCCARD=mCONFIG_PCMCIA=mCONFIG_PCMCIA_LOAD_CIS=yCONFIG_CARDBUS=y
#
# PC-card bridges
#CONFIG_YENTA=mCONFIG_YENTA_O2=yCONFIG_YENTA_RICOH=yCONFIG_YENTA_TI=yCONFIG_YENTA_ENE_TUNE=yCONFIG_YENTA_TOSHIBA=yCONFIG_PD6729=mCONFIG_I82092=mCONFIG_PCCARD_NONSTATIC=yCONFIG_RAPIDIO=yCONFIG_RAPIDIO_TSI721=mCONFIG_RAPIDIO_DISC_TIMEOUT=30
# CONFIG_RAPIDIO_ENABLE_RX_TX_PORTS is not setCONFIG_RAPIDIO_DMA_ENGINE=y
# CONFIG_RAPIDIO_DEBUG is not setCONFIG_RAPIDIO_ENUM_BASIC=mCONFIG_RAPIDIO_CHMAN=mCONFIG_RAPIDIO_MPORT_CDEV=m
#
# RapidIO Switch drivers
#CONFIG_RAPIDIO_CPS_XX=mCONFIG_RAPIDIO_CPS_GEN2=mCONFIG_RAPIDIO_RXS_GEN3=m
# end of RapidIO Switch drivers
#
# Generic Driver Options
#CONFIG_AUXILIARY_BUS=yCONFIG_UEVENT_HELPER=yCONFIG_UEVENT_HELPER_PATH=""CONFIG_DEVTMPFS=yCONFIG_DEVTMPFS_MOUNT=y
# CONFIG_DEVTMPFS_SAFE is not set
# CONFIG_STANDALONE is not setCONFIG_PREVENT_FIRMWARE_BUILD=y
#
# Firmware loader
#CONFIG_FW_LOADER=y CONFIG_FW_LOADER_DEBUG=y CONFIG_FW_LOADER_PAGED_BUF=y CONFIG_FW_LOADER_SYSFS=y CONFIG_EXTRA_FIRMWARE="" CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set CONFIG_FW_LOADER_COMPRESS=y CONFIG_FW_LOADER_COMPRESS_XZ=y
# CONFIG_FW_LOADER_COMPRESS_ZSTD is not set CONFIG_FW_CACHE=yCONFIG_FW_UPLOAD=y
# end of Firmware loader
CONFIG_WANT_DEV_COREDUMP=yCONFIG_ALLOW_DEV_COREDUMP=yCONFIG_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not setCONFIG_HMEM_REPORTING=y
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not setCONFIG_SYS_HYPERVISOR=yCONFIG_GENERIC_CPU_DEVICES=yCONFIG_GENERIC_CPU_AUTOPROBE=yCONFIG_GENERIC_CPU_VULNERABILITIES=yCONFIG_REGMAP=yCONFIG_REGMAP_I2C=yCONFIG_REGMAP_SLIMBUS=mCONFIG_REGMAP_SPI=yCONFIG_REGMAP_SPMI=mCONFIG_REGMAP_W1=mCONFIG_REGMAP_MMIO=yCONFIG_REGMAP_IRQ=yCONFIG_REGMAP_SOUNDWIRE=mCONFIG_REGMAP_SOUNDWIRE_MBQ=mCONFIG_REGMAP_SCCB=mCONFIG_REGMAP_I3C=mCONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set
# CONFIG_FW_DEVLINK_SYNC_STATE_TIMEOUT is not set
# end of Generic Driver Options
#
# Bus devices
#CONFIG_MHI_BUS=m
# CONFIG_MHI_BUS_DEBUG is not setCONFIG_MHI_BUS_PCI_GENERIC=m
# CONFIG_MHI_BUS_EP is not set
# end of Bus devices
#
# Cache Drivers
#
# end of Cache Drivers
CONFIG_CONNECTOR=yCONFIG_PROC_EVENTS=y
#
# Firmware Drivers
#
#
# ARM System Control and Management Interface Protocol #
# end of ARM System Control and Management Interface Protocol
CONFIG_EDD=yCONFIG_EDD_OFF=yCONFIG_FIRMWARE_MEMMAP=yCONFIG_DMIID=yCONFIG_DMI_SYSFS=mCONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=yCONFIG_ISCSI_IBFT_FIND=yCONFIG_ISCSI_IBFT=mCONFIG_FW_CFG_SYSFS=m
# CONFIG_FW_CFG_SYSFS_CMDLINE is not setCONFIG_SYSFB=y
# CONFIG_SYSFB_SIMPLEFB is not setCONFIG_FW_CS_DSP=m
# CONFIG_GOOGLE_FIRMWARE is not set
#
# EFI (Extensible Firmware Interface) Support #CONFIG_EFI_ESRT=yCONFIG_EFI_VARS_PSTORE=m
# CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set CONFIG_EFI_SOFT_RESERVE=y CONFIG_EFI_DXE_MEM_ATTRIBUTES=y CONFIG_EFI_RUNTIME_WRAPPERS=y CONFIG_EFI_BOOTLOADER_CONTROL=m CONFIG_EFI_CAPSULE_LOADER=m CONFIG_EFI_TEST=m CONFIG_EFI_DEV_PATH_PARSER=y CONFIG_APPLE_PROPERTIES=y CONFIG_RESET_ATTACK_MITIGATION=y CONFIG_EFI_RCI2_TABLE=y
# CONFIG_EFI_DISABLE_PCI_DMA is not set CONFIG_EFI_EARLYCON=y CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
# CONFIG_EFI_DISABLE_RUNTIME is not set
# CONFIG_EFI_COCO_SECRET is not set CONFIG_UNACCEPTED_MEMORY=y CONFIG_EFI_EMBEDDED_FIRMWARE=y
# end of EFI (Extensible Firmware Interface) Support
CONFIG_UEFI_CPER=yCONFIG_UEFI_CPER_X86=y
#
# Qualcomm firmware drivers
#
# end of Qualcomm firmware drivers
#
# Tegra firmware driver
#
# end of Tegra firmware driver # end of Firmware Drivers
CONFIG_GNSS=mCONFIG_GNSS_SERIAL=mCONFIG_GNSS_MTK_SERIAL=mCONFIG_GNSS_SIRF_SERIAL=mCONFIG_GNSS_UBX_SERIAL=m
# CONFIG_GNSS_USB is not setCONFIG_MTD=m
# CONFIG_MTD_TESTS is not set
#
# Partition parsers
#CONFIG_MTD_CMDLINE_PARTS=mCONFIG_MTD_REDBOOT_PARTS=mCONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
# end of Partition parsers
#
# User Modules And Translation Layers
#CONFIG_MTD_BLKDEVS=mCONFIG_MTD_BLOCK=mCONFIG_MTD_BLOCK_RO=m
#
# Note that in some cases UBI block is preferred. See MTD_UBI_BLOCK. #CONFIG_FTL=mCONFIG_NFTL=mCONFIG_NFTL_RW=yCONFIG_INFTL=mCONFIG_RFD_FTL=mCONFIG_SSFDC=mCONFIG_SM_FTL=mCONFIG_MTD_OOPS=mCONFIG_MTD_PSTORE=mCONFIG_MTD_SWAP=m
# CONFIG_MTD_PARTITIONED_MASTER is not set
#
# RAM/ROM/Flash chip drivers
#CONFIG_MTD_CFI=mCONFIG_MTD_JEDECPROBE=mCONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not setCONFIG_MTD_MAP_BANK_WIDTH_1=yCONFIG_MTD_MAP_BANK_WIDTH_2=yCONFIG_MTD_MAP_BANK_WIDTH_4=yCONFIG_MTD_CFI_I1=yCONFIG_MTD_CFI_I2=yCONFIG_MTD_CFI_INTELEXT=mCONFIG_MTD_CFI_AMDSTD=mCONFIG_MTD_CFI_STAA=mCONFIG_MTD_CFI_UTIL=mCONFIG_MTD_RAM=mCONFIG_MTD_ROM=mCONFIG_MTD_ABSENT=m
# end of RAM/ROM/Flash chip drivers
#
# Mapping drivers for chip access
#CONFIG_MTD_COMPLEX_MAPPINGS=yCONFIG_MTD_PHYSMAP=m
# CONFIG_MTD_PHYSMAP_COMPAT is not setCONFIG_MTD_PHYSMAP_GPIO_ADDR=yCONFIG_MTD_SBC_GXX=mCONFIG_MTD_AMD76XROM=mCONFIG_MTD_ICHXROM=mCONFIG_MTD_ESB2ROM=mCONFIG_MTD_CK804XROM=mCONFIG_MTD_SCB2_FLASH=m
CONFIG_MTD_NETtel=mCONFIG_MTD_L440GX=mCONFIG_MTD_PCI=mCONFIG_MTD_PCMCIA=m
# CONFIG_MTD_PCMCIA_ANONYMOUS is not setCONFIG_MTD_PLATRAM=m
# end of Mapping drivers for chip access
#
# Self-contained MTD device drivers
#CONFIG_MTD_PMC551=m
# CONFIG_MTD_PMC551_BUGFIX is not set
# CONFIG_MTD_PMC551_DEBUG is not setCONFIG_MTD_DATAFLASH=m
# CONFIG_MTD_DATAFLASH_WRITE_VERIFY is not setCONFIG_MTD_DATAFLASH_OTP=yCONFIG_MTD_MCHP23K256=mCONFIG_MTD_MCHP48L640=mCONFIG_MTD_SST25L=mCONFIG_MTD_SLRAM=mCONFIG_MTD_PHRAM=mCONFIG_MTD_MTDRAM=mCONFIG_MTDRAM_TOTAL_SIZE=4096CONFIG_MTDRAM_ERASE_SIZE=128CONFIG_MTD_BLOCK2MTD=m
#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOCG3 is not set
# end of Self-contained MTD device drivers
#
# NAND
#CONFIG_MTD_NAND_CORE=mCONFIG_MTD_ONENAND=mCONFIG_MTD_ONENAND_VERIFY_WRITE=yCONFIG_MTD_ONENAND_GENERIC=m
# CONFIG_MTD_ONENAND_OTP is not setCONFIG_MTD_ONENAND_2X_PROGRAM=yCONFIG_MTD_RAW_NAND=m
#
# Raw/parallel NAND flash controllers
#CONFIG_MTD_NAND_DENALI=mCONFIG_MTD_NAND_DENALI_PCI=mCONFIG_MTD_NAND_CAFE=mCONFIG_MTD_NAND_MXIC=mCONFIG_MTD_NAND_GPIO=mCONFIG_MTD_NAND_PLATFORM=mCONFIG_MTD_NAND_ARASAN=m
#
# Misc
#CONFIG_MTD_SM_COMMON=m CONFIG_MTD_NAND_NANDSIM=m CONFIG_MTD_NAND_RICOH=m CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set CONFIG_MTD_SPI_NAND=m
#
# ECC engine support
#CONFIG_MTD_NAND_ECC=yCONFIG_MTD_NAND_ECC_SW_HAMMING=y
# CONFIG_MTD_NAND_ECC_SW_HAMMING_SMC is not setCONFIG_MTD_NAND_ECC_SW_BCH=yCONFIG_MTD_NAND_ECC_MXIC=y
# end of ECC engine support # end of NAND
#
# LPDDR & LPDDR2 PCM memory drivers
#CONFIG_MTD_LPDDR=mCONFIG_MTD_QINFO_PROBE=m
# end of LPDDR & LPDDR2 PCM memory drivers
CONFIG_MTD_SPI_NOR=mCONFIG_MTD_SPI_NOR_USE_4K_SECTORS=y
# CONFIG_MTD_SPI_NOR_SWP_DISABLE is not setCONFIG_MTD_SPI_NOR_SWP_DISABLE_ON_VOLATILE=y
# CONFIG_MTD_SPI_NOR_SWP_KEEP is not setCONFIG_MTD_UBI=mCONFIG_MTD_UBI_WL_THRESHOLD=4096CONFIG_MTD_UBI_BEB_LIMIT=20CONFIG_MTD_UBI_FASTMAP=yCONFIG_MTD_UBI_GLUEBI=mCONFIG_MTD_UBI_BLOCK=y
# CONFIG_MTD_UBI_NVMEM is not setCONFIG_MTD_HYPERBUS=m
# CONFIG_OF is not setCONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=yCONFIG_PARPORT=mCONFIG_PARPORT_PC=mCONFIG_PARPORT_SERIAL=mCONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not setCONFIG_PARPORT_PC_PCMCIA=mCONFIG_PARPORT_1284=yCONFIG_PARPORT_NOT_PC=yCONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
#
# Protocols
#CONFIG_PNPACPI=yCONFIG_BLK_DEV=yCONFIG_BLK_DEV_NULL_BLK=mCONFIG_BLK_DEV_FD=m
# CONFIG_BLK_DEV_FD_RAWCMD is not setCONFIG_CDROM=yCONFIG_BLK_DEV_PCIESSD_MTIP32XX=mCONFIG_ZRAM=mCONFIG_ZRAM_DEF_COMP_LZORLE=y
# CONFIG_ZRAM_DEF_COMP_ZSTD is not set
# CONFIG_ZRAM_DEF_COMP_LZ4 is not set
# CONFIG_ZRAM_DEF_COMP_LZO is not set
# CONFIG_ZRAM_DEF_COMP_LZ4HC is not set
# CONFIG_ZRAM_DEF_COMP_842 is not setCONFIG_ZRAM_DEF_COMP="lzo-rle"CONFIG_ZRAM_WRITEBACK=yCONFIG_ZRAM_TRACK_ENTRY_ACTIME=yCONFIG_ZRAM_MEMORY_TRACKING=y
# CONFIG_ZRAM_MULTI_COMP is not setCONFIG_BLK_DEV_LOOP=yCONFIG_BLK_DEV_LOOP_MIN_COUNT=8CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not setCONFIG_BLK_DEV_NBD=mCONFIG_BLK_DEV_RAM=mCONFIG_BLK_DEV_RAM_COUNT=16CONFIG_BLK_DEV_RAM_SIZE=65536CONFIG_CDROM_PKTCDVD=mCONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not setCONFIG_ATA_OVER_ETH=mCONFIG_XEN_BLKDEV_FRONTEND=yCONFIG_XEN_BLKDEV_BACKEND=mCONFIG_VIRTIO_BLK=mCONFIG_BLK_DEV_RBD=m
# CONFIG_BLK_DEV_UBLK is not setCONFIG_BLK_DEV_RNBD=yCONFIG_BLK_DEV_RNBD_CLIENT=mCONFIG_BLK_DEV_RNBD_SERVER=m
#
# NVME Support
#CONFIG_NVME_CORE=yCONFIG_BLK_DEV_NVME=yCONFIG_NVME_MULTIPATH=y
# CONFIG_NVME_VERBOSE_ERRORS is not setCONFIG_NVME_HWMON=yCONFIG_NVME_FABRICS=mCONFIG_NVME_RDMA=mCONFIG_NVME_FC=mCONFIG_NVME_TCP=m
# CONFIG_NVME_TCP_TLS is not set
# CONFIG_NVME_HOST_AUTH is not setCONFIG_NVME_TARGET=mCONFIG_NVME_TARGET_PASSTHRU=yCONFIG_NVME_TARGET_LOOP=mCONFIG_NVME_TARGET_RDMA=mCONFIG_NVME_TARGET_FC=m
# CONFIG_NVME_TARGET_FCLOOP is not setCONFIG_NVME_TARGET_TCP=m
# CONFIG_NVME_TARGET_TCP_TLS is not set
# CONFIG_NVME_TARGET_AUTH is not set
# end of NVME Support
#
# Misc devices
#CONFIG_SENSORS_LIS3LV02D=mCONFIG_AD525X_DPOT=mCONFIG_AD525X_DPOT_I2C=mCONFIG_AD525X_DPOT_SPI=mCONFIG_DUMMY_IRQ=mCONFIG_IBM_ASM=mCONFIG_PHANTOM=mCONFIG_TIFM_CORE=mCONFIG_TIFM_7XX1=mCONFIG_ICS932S401=mCONFIG_ENCLOSURE_SERVICES=mCONFIG_SGI_XP=mCONFIG_HP_ILO=mCONFIG_SGI_GRU=m
# CONFIG_SGI_GRU_DEBUG is not setCONFIG_APDS9802ALS=mCONFIG_ISL29003=mCONFIG_ISL29020=mCONFIG_SENSORS_TSL2550=mCONFIG_SENSORS_BH1770=mCONFIG_SENSORS_APDS990X=mCONFIG_HMC6352=mCONFIG_DS1682=mCONFIG_VMWARE_BALLOON=mCONFIG_LATTICE_ECP3_CONFIG=mCONFIG_SRAM=yCONFIG_DW_XDATA_PCIE=m
# CONFIG_PCI_ENDPOINT_TEST is not setCONFIG_XILINX_SDFEC=mCONFIG_MISC_RTSX=m
# CONFIG_NSM is not setCONFIG_C2PORT=mCONFIG_C2PORT_DURAMAR_2150=m
#
# EEPROM support
#CONFIG_EEPROM_AT24=mCONFIG_EEPROM_AT25=mCONFIG_EEPROM_MAX6875=mCONFIG_EEPROM_93CX6=mCONFIG_EEPROM_93XX46=mCONFIG_EEPROM_IDT_89HPESX=mCONFIG_EEPROM_EE1004=m
# end of EEPROM support
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not setCONFIG_CB710_DEBUG_ASSUMPTIONS=y
#
# Texas Instruments shared transport line discipline #CONFIG_TI_ST=m
# end of Texas Instruments shared transport line discipline
CONFIG_SENSORS_LIS3_I2C=mCONFIG_ALTERA_STAPL=mCONFIG_INTEL_MEI=mCONFIG_INTEL_MEI_ME=mCONFIG_INTEL_MEI_TXE=m
# CONFIG_INTEL_MEI_GSC is not set
# CONFIG_INTEL_MEI_VSC_HW is not setCONFIG_INTEL_MEI_HDCP=m
# CONFIG_INTEL_MEI_PXP is not set
# CONFIG_INTEL_MEI_GSC_PROXY is not setCONFIG_VMWARE_VMCI=mCONFIG_GENWQE=mCONFIG_GENWQE_PLATFORM_ERROR_RECOVERY=0CONFIG_ECHO=mCONFIG_BCM_VK=mCONFIG_BCM_VK_TTY=yCONFIG_MISC_ALCOR_PCI=mCONFIG_MISC_RTSX_PCI=mCONFIG_MISC_RTSX_USB=mCONFIG_UACCE=mCONFIG_PVPANIC=yCONFIG_PVPANIC_MMIO=mCONFIG_PVPANIC_PCI=m
# CONFIG_GP_PCI1XXXX is not set
# end of Misc devices
#
# SCSI device support
#CONFIG_SCSI_MOD=yCONFIG_RAID_ATTRS=mCONFIG_SCSI_COMMON=yCONFIG_SCSI=yCONFIG_SCSI_DMA=yCONFIG_SCSI_NETLINK=yCONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#CONFIG_BLK_DEV_SD=yCONFIG_CHR_DEV_ST=mCONFIG_BLK_DEV_SR=yCONFIG_CHR_DEV_SG=yCONFIG_BLK_DEV_BSG=yCONFIG_CHR_DEV_SCH=mCONFIG_SCSI_ENCLOSURE=mCONFIG_SCSI_CONSTANTS=yCONFIG_SCSI_LOGGING=yCONFIG_SCSI_SCAN_ASYNC=y
#
# SCSI Transports
#CONFIG_SCSI_SPI_ATTRS=mCONFIG_SCSI_FC_ATTRS=mCONFIG_SCSI_ISCSI_ATTRS=mCONFIG_SCSI_SAS_ATTRS=mCONFIG_SCSI_SAS_LIBSAS=mCONFIG_SCSI_SAS_ATA=yCONFIG_SCSI_SAS_HOST_SMP=yCONFIG_SCSI_SRP_ATTRS=m
# end of SCSI Transports
CONFIG_SCSI_LOWLEVEL=yCONFIG_ISCSI_TCP=mCONFIG_ISCSI_BOOT_SYSFS=mCONFIG_SCSI_CXGB3_ISCSI=mCONFIG_SCSI_CXGB4_ISCSI=mCONFIG_SCSI_BNX2_ISCSI=mCONFIG_SCSI_BNX2X_FCOE=mCONFIG_BE2ISCSI=mCONFIG_BLK_DEV_3W_XXXX_RAID=mCONFIG_SCSI_HPSA=mCONFIG_SCSI_3W_9XXX=mCONFIG_SCSI_3W_SAS=mCONFIG_SCSI_ACARD=mCONFIG_SCSI_AHA1740=mCONFIG_SCSI_AACRAID=mCONFIG_SCSI_AIC7XXX=mCONFIG_AIC7XXX_CMDS_PER_DEVICE=8CONFIG_AIC7XXX_RESET_DELAY_MS=5000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not setCONFIG_AIC7XXX_DEBUG_MASK=0CONFIG_AIC7XXX_REG_PRETTY_PRINT=yCONFIG_SCSI_AIC79XX=mCONFIG_AIC79XX_CMDS_PER_DEVICE=32CONFIG_AIC79XX_RESET_DELAY_MS=5000
# CONFIG_AIC79XX_DEBUG_ENABLE is not setCONFIG_AIC79XX_DEBUG_MASK=0CONFIG_AIC79XX_REG_PRETTY_PRINT=yCONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not setCONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
# CONFIG_SCSI_MVSAS_TASKLET is not setCONFIG_SCSI_MVUMI=mCONFIG_SCSI_ADVANSYS=mCONFIG_SCSI_ARCMSR=mCONFIG_SCSI_ESAS2R=mCONFIG_MEGARAID_NEWGEN=yCONFIG_MEGARAID_MM=mCONFIG_MEGARAID_MAILBOX=mCONFIG_MEGARAID_LEGACY=mCONFIG_MEGARAID_SAS=mCONFIG_SCSI_MPT3SAS=mCONFIG_SCSI_MPT2SAS_MAX_SGE=128CONFIG_SCSI_MPT3SAS_MAX_SGE=128CONFIG_SCSI_MPT2SAS=mCONFIG_SCSI_MPI3MR=mCONFIG_SCSI_SMARTPQI=mCONFIG_SCSI_HPTIOP=mCONFIG_SCSI_BUSLOGIC=mCONFIG_SCSI_FLASHPOINT=yCONFIG_SCSI_MYRB=mCONFIG_SCSI_MYRS=mCONFIG_VMWARE_PVSCSI=mCONFIG_XEN_SCSI_FRONTEND=mCONFIG_HYPERV_STORAGE=mCONFIG_LIBFC=mCONFIG_LIBFCOE=mCONFIG_FCOE=mCONFIG_FCOE_FNIC=mCONFIG_SCSI_SNIC=m
# CONFIG_SCSI_SNIC_DEBUG_FS is not setCONFIG_SCSI_DMX3191D=mCONFIG_SCSI_FDOMAIN=mCONFIG_SCSI_FDOMAIN_PCI=mCONFIG_SCSI_ISCI=mCONFIG_SCSI_IPS=mCONFIG_SCSI_INITIO=mCONFIG_SCSI_INIA100=mCONFIG_SCSI_PPA=mCONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_SLOW_CTR is not setCONFIG_SCSI_STEX=mCONFIG_SCSI_SYM53C8XX_2=mCONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64CONFIG_SCSI_SYM53C8XX_MMIO=yCONFIG_SCSI_IPR=mCONFIG_SCSI_IPR_TRACE=yCONFIG_SCSI_IPR_DUMP=yCONFIG_SCSI_QLOGIC_1280=mCONFIG_SCSI_QLA_FC=mCONFIG_TCM_QLA2XXX=m
# CONFIG_TCM_QLA2XXX_DEBUG is not setCONFIG_SCSI_QLA_ISCSI=mCONFIG_QEDI=mCONFIG_QEDF=mCONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not setCONFIG_SCSI_EFCT=mCONFIG_SCSI_SIM710=m
CONFIG_SCSI_DC395x=mCONFIG_SCSI_AM53C974=mCONFIG_SCSI_WD719X=mCONFIG_SCSI_DEBUG=mCONFIG_SCSI_PMCRAID=mCONFIG_SCSI_PM8001=mCONFIG_SCSI_BFA_FC=mCONFIG_SCSI_VIRTIO=mCONFIG_SCSI_CHELSIO_FCOE=mCONFIG_SCSI_LOWLEVEL_PCMCIA=yCONFIG_PCMCIA_AHA152X=mCONFIG_PCMCIA_FDOMAIN=mCONFIG_PCMCIA_QLOGIC=mCONFIG_PCMCIA_SYM53C500=mCONFIG_SCSI_DH=yCONFIG_SCSI_DH_RDAC=mCONFIG_SCSI_DH_HP_SW=mCONFIG_SCSI_DH_EMC=mCONFIG_SCSI_DH_ALUA=m
# end of SCSI device support
CONFIG_ATA=yCONFIG_SATA_HOST=yCONFIG_PATA_TIMINGS=yCONFIG_ATA_VERBOSE_ERROR=yCONFIG_ATA_FORCE=yCONFIG_ATA_ACPI=yCONFIG_SATA_ZPODD=yCONFIG_SATA_PMP=y
#
# Controllers with non-SFF native interface
#CONFIG_SATA_AHCI=yCONFIG_SATA_MOBILE_LPM_POLICY=3CONFIG_SATA_AHCI_PLATFORM=m
# CONFIG_AHCI_DWC is not setCONFIG_SATA_INIC162X=mCONFIG_SATA_ACARD_AHCI=mCONFIG_SATA_SIL24=mCONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#CONFIG_PDC_ADMA=mCONFIG_SATA_QSTOR=mCONFIG_SATA_SX4=mCONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#CONFIG_ATA_PIIX=yCONFIG_SATA_DWC=mCONFIG_SATA_DWC_OLD_DMA=yCONFIG_SATA_MV=mCONFIG_SATA_NV=mCONFIG_SATA_PROMISE=mCONFIG_SATA_SIL=mCONFIG_SATA_SIS=mCONFIG_SATA_SVW=mCONFIG_SATA_ULI=mCONFIG_SATA_VIA=mCONFIG_SATA_VITESSE=m
#
# PATA SFF controllers with BMDMA
#CONFIG_PATA_ALI=mCONFIG_PATA_AMD=mCONFIG_PATA_ARTOP=mCONFIG_PATA_ATIIXP=mCONFIG_PATA_ATP867X=mCONFIG_PATA_CMD64X=mCONFIG_PATA_CYPRESS=mCONFIG_PATA_EFAR=mCONFIG_PATA_HPT366=mCONFIG_PATA_HPT37X=mCONFIG_PATA_HPT3X2N=mCONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not setCONFIG_PATA_IT8213=mCONFIG_PATA_IT821X=mCONFIG_PATA_JMICRON=mCONFIG_PATA_MARVELL=mCONFIG_PATA_NETCELL=mCONFIG_PATA_NINJA32=mCONFIG_PATA_NS87415=mCONFIG_PATA_OLDPIIX=mCONFIG_PATA_OPTIDMA=mCONFIG_PATA_PDC2027X=mCONFIG_PATA_PDC_OLD=mCONFIG_PATA_RADISYS=mCONFIG_PATA_RDC=mCONFIG_PATA_SCH=mCONFIG_PATA_SERVERWORKS=mCONFIG_PATA_SIL680=mCONFIG_PATA_SIS=yCONFIG_PATA_TOSHIBA=mCONFIG_PATA_TRIFLEX=mCONFIG_PATA_VIA=mCONFIG_PATA_WINBOND=m
#
# PIO-only SFF controllers
#CONFIG_PATA_CMD640_PCI=mCONFIG_PATA_MPIIX=mCONFIG_PATA_NS87410=mCONFIG_PATA_OPTI=mCONFIG_PATA_PCMCIA=mCONFIG_PATA_RZ1000=m
# CONFIG_PATA_PARPORT is not set
#
# Generic fallback / legacy drivers
#CONFIG_PATA_ACPI=mCONFIG_ATA_GENERIC=yCONFIG_PATA_LEGACY=mCONFIG_MD=yCONFIG_BLK_DEV_MD=yCONFIG_MD_AUTODETECT=yCONFIG_MD_BITMAP_FILE=yCONFIG_MD_RAID0=mCONFIG_MD_RAID1=mCONFIG_MD_RAID10=mCONFIG_MD_RAID456=mCONFIG_MD_CLUSTER=mCONFIG_BCACHE=m
# CONFIG_BCACHE_DEBUG is not set CONFIG_BCACHE_ASYNC_REGISTRATION=y CONFIG_BLK_DEV_DM_BUILTIN=yCONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not setCONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set CONFIG_DM_BIO_PRISON=mCONFIG_DM_PERSISTENT_DATA=mCONFIG_DM_UNSTRIPED=mCONFIG_DM_CRYPT=mCONFIG_DM_SNAPSHOT=mCONFIG_DM_THIN_PROVISIONING=mCONFIG_DM_CACHE=mCONFIG_DM_CACHE_SMQ=mCONFIG_DM_WRITECACHE=mCONFIG_DM_EBS=mCONFIG_DM_ERA=mCONFIG_DM_CLONE=mCONFIG_DM_MIRROR=mCONFIG_DM_LOG_USERSPACE=mCONFIG_DM_RAID=mCONFIG_DM_ZERO=mCONFIG_DM_MULTIPATH=mCONFIG_DM_MULTIPATH_QL=mCONFIG_DM_MULTIPATH_ST=mCONFIG_DM_MULTIPATH_HST=mCONFIG_DM_MULTIPATH_IOA=mCONFIG_DM_DELAY=m
# CONFIG_DM_DUST is not setCONFIG_DM_INIT=yCONFIG_DM_UEVENT=yCONFIG_DM_FLAKEY=mCONFIG_DM_VERITY=m CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG=y
# CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG_SECONDARY_KEYRING is not set # CONFIG_DM_VERITY_FEC is not setCONFIG_DM_SWITCH=mCONFIG_DM_LOG_WRITES=mCONFIG_DM_INTEGRITY=mCONFIG_DM_ZONED=mCONFIG_DM_AUDIT=y
# CONFIG_DM_VDO is not setCONFIG_TARGET_CORE=mCONFIG_TCM_IBLOCK=mCONFIG_TCM_FILEIO=mCONFIG_TCM_PSCSI=mCONFIG_TCM_USER2=mCONFIG_LOOPBACK_TARGET=mCONFIG_TCM_FC=mCONFIG_ISCSI_TARGET=mCONFIG_ISCSI_TARGET_CXGB4=mCONFIG_SBP_TARGET=m
# CONFIG_REMOTE_TARGET is not setCONFIG_FUSION=yCONFIG_FUSION_SPI=mCONFIG_FUSION_FC=mCONFIG_FUSION_SAS=mCONFIG_FUSION_MAX_SGE=128CONFIG_FUSION_CTL=mCONFIG_FUSION_LAN=mCONFIG_FUSION_LOGGING=y
#
# IEEE 1394 (FireWire) support
#CONFIG_FIREWIRE=mCONFIG_FIREWIRE_OHCI=mCONFIG_FIREWIRE_SBP2=mCONFIG_FIREWIRE_NET=mCONFIG_FIREWIRE_NOSY=m
# end of IEEE 1394 (FireWire) support
CONFIG_MACINTOSH_DRIVERS=yCONFIG_MAC_EMUMOUSEBTN=mCONFIG_NETDEVICES=yCONFIG_MII=mCONFIG_NET_CORE=yCONFIG_BONDING=mCONFIG_DUMMY=mCONFIG_WIREGUARD=m
# CONFIG_WIREGUARD_DEBUG is not setCONFIG_EQUALIZER=mCONFIG_NET_FC=yCONFIG_IFB=mCONFIG_NET_TEAM=mCONFIG_NET_TEAM_MODE_BROADCAST=mCONFIG_NET_TEAM_MODE_ROUNDROBIN=mCONFIG_NET_TEAM_MODE_RANDOM=mCONFIG_NET_TEAM_MODE_ACTIVEBACKUP=mCONFIG_NET_TEAM_MODE_LOADBALANCE=mCONFIG_MACVLAN=mCONFIG_MACVTAP=mCONFIG_IPVLAN_L3S=yCONFIG_IPVLAN=mCONFIG_IPVTAP=mCONFIG_VXLAN=mCONFIG_GENEVE=mCONFIG_BAREUDP=mCONFIG_GTP=m
# CONFIG_AMT is not setCONFIG_MACSEC=mCONFIG_NETCONSOLE=mCONFIG_NETCONSOLE_DYNAMIC=y
# CONFIG_NETCONSOLE_EXTENDED_LOG is not setCONFIG_NETPOLL=yCONFIG_NET_POLL_CONTROLLER=yCONFIG_NTB_NETDEV=mCONFIG_RIONET=mCONFIG_RIONET_TX_SIZE=128CONFIG_RIONET_RX_SIZE=128CONFIG_TUN=yCONFIG_TAP=m
# CONFIG_TUN_VNET_CROSS_LE is not setCONFIG_VETH=mCONFIG_VIRTIO_NET=mCONFIG_NLMON=m
# CONFIG_NETKIT is not setCONFIG_NET_VRF=mCONFIG_VSOCKMON=mCONFIG_MHI_NET=mCONFIG_SUNGEM_PHY=mCONFIG_ARCNET=mCONFIG_ARCNET_1201=mCONFIG_ARCNET_1051=mCONFIG_ARCNET_RAW=mCONFIG_ARCNET_CAP=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=mCONFIG_ARCNET_RIM_I=mCONFIG_ARCNET_COM20020=mCONFIG_ARCNET_COM20020_PCI=mCONFIG_ARCNET_COM20020_CS=mCONFIG_ATM_DRIVERS=yCONFIG_ATM_DUMMY=mCONFIG_ATM_TCP=mCONFIG_ATM_LANAI=mCONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not setCONFIG_ATM_NICSTAR=m
# CONFIG_ATM_NICSTAR_USE_SUNI is not set
# CONFIG_ATM_NICSTAR_USE_IDT77105 is not setCONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not setCONFIG_ATM_IDT77252_USE_SUNI=yCONFIG_ATM_IA=m
# CONFIG_ATM_IA_DEBUG is not setCONFIG_ATM_FORE200E=m
# CONFIG_ATM_FORE200E_USE_TASKLET is not setCONFIG_ATM_FORE200E_TX_RETRY=16CONFIG_ATM_FORE200E_DEBUG=0CONFIG_ATM_HE=mCONFIG_ATM_HE_USE_SUNI=yCONFIG_ATM_SOLOS=mCONFIG_CAIF_DRIVERS=yCONFIG_CAIF_TTY=mCONFIG_CAIF_VIRTIO=m
#
# Distributed Switch Architecture drivers
#CONFIG_B53=mCONFIG_B53_SPI_DRIVER=mCONFIG_B53_MDIO_DRIVER=mCONFIG_B53_MMAP_DRIVER=mCONFIG_B53_SRAB_DRIVER=mCONFIG_B53_SERDES=mCONFIG_NET_DSA_BCM_SF2=m
# CONFIG_NET_DSA_LOOP is not setCONFIG_NET_DSA_HIRSCHMANN_HELLCREEK=mCONFIG_NET_DSA_LANTIQ_GSWIP=mCONFIG_NET_DSA_MT7530=mCONFIG_NET_DSA_MT7530_MDIO=mCONFIG_NET_DSA_MT7530_MMIO=mCONFIG_NET_DSA_MV88E6060=mCONFIG_NET_DSA_MICROCHIP_KSZ_COMMON=mCONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C=m
# CONFIG_NET_DSA_MICROCHIP_KSZ_SPI is not set
# CONFIG_NET_DSA_MICROCHIP_KSZ_PTP is not setCONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI=mCONFIG_NET_DSA_MV88E6XXX=mCONFIG_NET_DSA_MV88E6XXX_PTP=yCONFIG_NET_DSA_MSCC_FELIX_DSA_LIB=m
# CONFIG_NET_DSA_MSCC_OCELOT_EXT is not setCONFIG_NET_DSA_MSCC_SEVILLE=mCONFIG_NET_DSA_AR9331=mCONFIG_NET_DSA_QCA8K=m
# CONFIG_NET_DSA_QCA8K_LEDS_SUPPORT is not setCONFIG_NET_DSA_SJA1105=mCONFIG_NET_DSA_SJA1105_PTP=yCONFIG_NET_DSA_SJA1105_TAS=yCONFIG_NET_DSA_SJA1105_VL=yCONFIG_NET_DSA_XRS700X=mCONFIG_NET_DSA_XRS700X_I2C=mCONFIG_NET_DSA_XRS700X_MDIO=m
# CONFIG_NET_DSA_REALTEK is not setCONFIG_NET_DSA_SMSC_LAN9303=mCONFIG_NET_DSA_SMSC_LAN9303_I2C=mCONFIG_NET_DSA_SMSC_LAN9303_MDIO=mCONFIG_NET_DSA_VITESSE_VSC73XX=mCONFIG_NET_DSA_VITESSE_VSC73XX_SPI=mCONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM=m
# end of Distributed Switch Architecture drivers
CONFIG_ETHERNET=yCONFIG_MDIO=mCONFIG_NET_VENDOR_3COM=yCONFIG_EL3=mCONFIG_PCMCIA_3C574=mCONFIG_PCMCIA_3C589=mCONFIG_VORTEX=mCONFIG_TYPHOON=mCONFIG_NET_VENDOR_ADAPTEC=yCONFIG_ADAPTEC_STARFIRE=mCONFIG_NET_VENDOR_AGERE=yCONFIG_ET131X=mCONFIG_NET_VENDOR_ALACRITECH=yCONFIG_SLICOSS=mCONFIG_NET_VENDOR_ALTEON=yCONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not setCONFIG_ALTERA_TSE=mCONFIG_NET_VENDOR_AMAZON=yCONFIG_ENA_ETHERNET=mCONFIG_NET_VENDOR_AMD=yCONFIG_AMD8111_ETH=mCONFIG_PCNET32=mCONFIG_PCMCIA_NMCLAN=mCONFIG_AMD_XGBE=mCONFIG_AMD_XGBE_DCB=yCONFIG_AMD_XGBE_HAVE_ECC=y
# CONFIG_PDS_CORE is not setCONFIG_NET_VENDOR_AQUANTIA=yCONFIG_AQTION=mCONFIG_NET_VENDOR_ARC=yCONFIG_NET_VENDOR_ASIX=y
# CONFIG_SPI_AX88796C is not setCONFIG_NET_VENDOR_ATHEROS=yCONFIG_ATL2=mCONFIG_ATL1=mCONFIG_ATL1E=mCONFIG_ATL1C=mCONFIG_ALX=mCONFIG_CX_ECAT=mCONFIG_NET_VENDOR_BROADCOM=yCONFIG_B44=mCONFIG_B44_PCI_AUTOSELECT=yCONFIG_B44_PCICORE_AUTOSELECT=yCONFIG_B44_PCI=yCONFIG_BCMGENET=mCONFIG_BNX2=mCONFIG_CNIC=mCONFIG_TIGON3=mCONFIG_TIGON3_HWMON=yCONFIG_BNX2X=mCONFIG_BNX2X_SRIOV=yCONFIG_SYSTEMPORT=mCONFIG_BNXT=mCONFIG_BNXT_SRIOV=yCONFIG_BNXT_FLOWER_OFFLOAD=yCONFIG_BNXT_DCB=yCONFIG_BNXT_HWMON=yCONFIG_NET_VENDOR_CADENCE=yCONFIG_MACB=mCONFIG_MACB_USE_HWSTAMP=yCONFIG_MACB_PCI=mCONFIG_NET_VENDOR_CAVIUM=yCONFIG_THUNDER_NIC_PF=mCONFIG_THUNDER_NIC_VF=mCONFIG_THUNDER_NIC_BGX=mCONFIG_THUNDER_NIC_RGX=mCONFIG_CAVIUM_PTP=mCONFIG_LIQUIDIO_CORE=mCONFIG_LIQUIDIO=mCONFIG_LIQUIDIO_VF=mCONFIG_NET_VENDOR_CHELSIO=yCONFIG_CHELSIO_T1=mCONFIG_CHELSIO_T1_1G=yCONFIG_CHELSIO_T3=mCONFIG_CHELSIO_T4=mCONFIG_CHELSIO_T4_DCB=yCONFIG_CHELSIO_T4_FCOE=yCONFIG_CHELSIO_T4VF=mCONFIG_CHELSIO_LIB=mCONFIG_CHELSIO_INLINE_CRYPTO=yCONFIG_CHELSIO_IPSEC_INLINE=mCONFIG_CHELSIO_TLS_DEVICE=mCONFIG_NET_VENDOR_CIRRUS=yCONFIG_NET_VENDOR_CISCO=yCONFIG_ENIC=mCONFIG_NET_VENDOR_CORTINA=yCONFIG_NET_VENDOR_DAVICOM=y
# CONFIG_DM9051 is not setCONFIG_DNET=mCONFIG_NET_VENDOR_DEC=yCONFIG_NET_TULIP=yCONFIG_DE2104X=mCONFIG_DE2104X_DSL=0CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
# CONFIG_TULIP_NAPI is not setCONFIG_WINBOND_840=mCONFIG_DM9102=mCONFIG_ULI526X=mCONFIG_PCMCIA_XIRCOM=mCONFIG_NET_VENDOR_DLINK=yCONFIG_DL2K=mCONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not setCONFIG_NET_VENDOR_EMULEX=yCONFIG_BE2NET=mCONFIG_BE2NET_HWMON=yCONFIG_BE2NET_BE2=yCONFIG_BE2NET_BE3=yCONFIG_BE2NET_LANCER=yCONFIG_BE2NET_SKYHAWK=yCONFIG_NET_VENDOR_ENGLEDER=y
# CONFIG_TSNEP is not setCONFIG_NET_VENDOR_EZCHIP=yCONFIG_NET_VENDOR_FUJITSU=yCONFIG_PCMCIA_FMVJ18X=mCONFIG_NET_VENDOR_FUNGIBLE=y
# CONFIG_FUN_ETH is not setCONFIG_NET_VENDOR_GOOGLE=yCONFIG_GVE=mCONFIG_NET_VENDOR_HUAWEI=yCONFIG_HINIC=mCONFIG_NET_VENDOR_I825XX=yCONFIG_NET_VENDOR_INTEL=yCONFIG_E100=mCONFIG_E1000=mCONFIG_E1000E=mCONFIG_E1000E_HWTS=yCONFIG_IGB=mCONFIG_IGB_HWMON=yCONFIG_IGB_DCA=yCONFIG_IGBVF=mCONFIG_IXGBE=mCONFIG_IXGBE_HWMON=yCONFIG_IXGBE_DCA=yCONFIG_IXGBE_DCB=yCONFIG_IXGBE_IPSEC=yCONFIG_IXGBEVF=mCONFIG_IXGBEVF_IPSEC=yCONFIG_I40E=mCONFIG_I40E_DCB=yCONFIG_IAVF=mCONFIG_I40EVF=mCONFIG_ICE=mCONFIG_ICE_HWMON=yCONFIG_ICE_SWITCHDEV=yCONFIG_ICE_HWTS=yCONFIG_FM10K=mCONFIG_IGC=mCONFIG_IGC_LEDS=y
# CONFIG_IDPF is not setCONFIG_JME=mCONFIG_NET_VENDOR_ADI=y
# CONFIG_ADIN1110 is not setCONFIG_NET_VENDOR_LITEX=yCONFIG_NET_VENDOR_MARVELL=yCONFIG_MVMDIO=mCONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not setCONFIG_SKGE_GENESIS=yCONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
# CONFIG_OCTEON_EP is not set
# CONFIG_OCTEON_EP_VF is not setCONFIG_PRESTERA=mCONFIG_PRESTERA_PCI=mCONFIG_NET_VENDOR_MELLANOX=yCONFIG_MLX4_EN=mCONFIG_MLX4_EN_DCB=yCONFIG_MLX4_CORE=mCONFIG_MLX4_DEBUG=yCONFIG_MLX4_CORE_GEN2=yCONFIG_MLX5_CORE=mCONFIG_MLX5_FPGA=yCONFIG_MLX5_CORE_EN=yCONFIG_MLX5_EN_ARFS=yCONFIG_MLX5_EN_RXNFC=yCONFIG_MLX5_MPFS=yCONFIG_MLX5_ESWITCH=yCONFIG_MLX5_BRIDGE=yCONFIG_MLX5_CLS_ACT=yCONFIG_MLX5_TC_CT=yCONFIG_MLX5_TC_SAMPLE=yCONFIG_MLX5_CORE_EN_DCB=yCONFIG_MLX5_CORE_IPOIB=y
# CONFIG_MLX5_MACSEC is not setCONFIG_MLX5_EN_IPSEC=yCONFIG_MLX5_EN_TLS=yCONFIG_MLX5_SW_STEERING=yCONFIG_MLX5_SF=yCONFIG_MLX5_SF_MANAGER=y
# CONFIG_MLX5_DPLL is not setCONFIG_MLXSW_CORE=mCONFIG_MLXSW_CORE_HWMON=yCONFIG_MLXSW_CORE_THERMAL=yCONFIG_MLXSW_PCI=mCONFIG_MLXSW_I2C=mCONFIG_MLXSW_SPECTRUM=mCONFIG_MLXSW_SPECTRUM_DCB=yCONFIG_MLXSW_MINIMAL=mCONFIG_MLXFW=mCONFIG_NET_VENDOR_MICREL=yCONFIG_KS8842=mCONFIG_KS8851=mCONFIG_KS8851_MLL=mCONFIG_KSZ884X_PCI=mCONFIG_NET_VENDOR_MICROCHIP=yCONFIG_ENC28J60=m
# CONFIG_ENC28J60_WRITEVERIFY is not setCONFIG_ENCX24J600=mCONFIG_LAN743X=m
# CONFIG_VCAP is not setCONFIG_NET_VENDOR_MICROSEMI=yCONFIG_MSCC_OCELOT_SWITCH_LIB=mCONFIG_NET_VENDOR_MICROSOFT=yCONFIG_MICROSOFT_MANA=mCONFIG_NET_VENDOR_MYRI=yCONFIG_MYRI10GE=mCONFIG_MYRI10GE_DCA=yCONFIG_FEALNX=mCONFIG_NET_VENDOR_NI=yCONFIG_NI_XGE_MANAGEMENT_ENET=mCONFIG_NET_VENDOR_NATSEMI=yCONFIG_NATSEMI=mCONFIG_NS83820=mCONFIG_NET_VENDOR_NETERION=yCONFIG_S2IO=mCONFIG_NET_VENDOR_NETRONOME=yCONFIG_NFP=mCONFIG_NFP_APP_FLOWER=yCONFIG_NFP_APP_ABM_NIC=yCONFIG_NFP_NET_IPSEC=y
# CONFIG_NFP_DEBUG is not setCONFIG_NET_VENDOR_8390=yCONFIG_PCMCIA_AXNET=mCONFIG_NE2K_PCI=mCONFIG_PCMCIA_PCNET=mCONFIG_NET_VENDOR_NVIDIA=yCONFIG_FORCEDETH=mCONFIG_NET_VENDOR_OKI=yCONFIG_ETHOC=mCONFIG_NET_VENDOR_PACKET_ENGINES=yCONFIG_HAMACHI=mCONFIG_YELLOWFIN=mCONFIG_NET_VENDOR_PENSANDO=yCONFIG_IONIC=mCONFIG_NET_VENDOR_QLOGIC=yCONFIG_QLA3XXX=mCONFIG_QLCNIC=mCONFIG_QLCNIC_SRIOV=yCONFIG_QLCNIC_DCB=yCONFIG_QLCNIC_HWMON=yCONFIG_NETXEN_NIC=mCONFIG_QED=mCONFIG_QED_LL2=yCONFIG_QED_SRIOV=yCONFIG_QEDE=mCONFIG_QED_RDMA=yCONFIG_QED_ISCSI=yCONFIG_QED_FCOE=yCONFIG_QED_OOO=yCONFIG_NET_VENDOR_BROCADE=yCONFIG_BNA=mCONFIG_NET_VENDOR_QUALCOMM=yCONFIG_QCOM_EMAC=mCONFIG_RMNET=mCONFIG_NET_VENDOR_RDC=yCONFIG_R6040=mCONFIG_NET_VENDOR_REALTEK=yCONFIG_ATP=mCONFIG_8139CP=mCONFIG_8139TOO=mCONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not setCONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not setCONFIG_R8169=mCONFIG_R8169_LEDS=yCONFIG_NET_VENDOR_RENESAS=yCONFIG_NET_VENDOR_ROCKER=yCONFIG_ROCKER=mCONFIG_NET_VENDOR_SAMSUNG=yCONFIG_SXGBE_ETH=mCONFIG_NET_VENDOR_SEEQ=yCONFIG_NET_VENDOR_SILAN=yCONFIG_SC92031=mCONFIG_NET_VENDOR_SIS=yCONFIG_SIS900=mCONFIG_SIS190=mCONFIG_NET_VENDOR_SOLARFLARE=yCONFIG_SFC=mCONFIG_SFC_MTD=yCONFIG_SFC_MCDI_MON=yCONFIG_SFC_SRIOV=yCONFIG_SFC_MCDI_LOGGING=yCONFIG_SFC_FALCON=mCONFIG_SFC_FALCON_MTD=y
# CONFIG_SFC_SIENA is not setCONFIG_NET_VENDOR_SMSC=yCONFIG_PCMCIA_SMC91C92=mCONFIG_EPIC100=mCONFIG_SMSC911X=mCONFIG_SMSC9420=mCONFIG_NET_VENDOR_SOCIONEXT=yCONFIG_NET_VENDOR_STMICRO=yCONFIG_STMMAC_ETH=m
# CONFIG_STMMAC_SELFTESTS is not setCONFIG_STMMAC_PLATFORM=mCONFIG_DWMAC_GENERIC=mCONFIG_DWMAC_INTEL=mCONFIG_STMMAC_PCI=mCONFIG_NET_VENDOR_SUN=yCONFIG_HAPPYMEAL=mCONFIG_SUNGEM=mCONFIG_CASSINI=mCONFIG_NIU=mCONFIG_NET_VENDOR_SYNOPSYS=yCONFIG_DWC_XLGMAC=mCONFIG_DWC_XLGMAC_PCI=mCONFIG_NET_VENDOR_TEHUTI=yCONFIG_TEHUTI=mCONFIG_NET_VENDOR_TI=y
# CONFIG_TI_CPSW_PHY_SEL is not setCONFIG_TLAN=mCONFIG_NET_VENDOR_VERTEXCOM=y
# CONFIG_MSE102X is not setCONFIG_NET_VENDOR_VIA=yCONFIG_VIA_RHINE=mCONFIG_VIA_RHINE_MMIO=yCONFIG_VIA_VELOCITY=mCONFIG_NET_VENDOR_WANGXUN=y
# CONFIG_NGBE is not set
# CONFIG_TXGBE is not setCONFIG_NET_VENDOR_WIZNET=yCONFIG_WIZNET_W5100=mCONFIG_WIZNET_W5300=m
# CONFIG_WIZNET_BUS_DIRECT is not set
# CONFIG_WIZNET_BUS_INDIRECT is not setCONFIG_WIZNET_BUS_ANY=yCONFIG_WIZNET_W5100_SPI=mCONFIG_NET_VENDOR_XILINX=yCONFIG_XILINX_EMACLITE=mCONFIG_XILINX_LL_TEMAC=mCONFIG_NET_VENDOR_XIRCOM=yCONFIG_PCMCIA_XIRC2PS=mCONFIG_FDDI=yCONFIG_DEFXX=mCONFIG_SKFP=m
# CONFIG_HIPPI is not setCONFIG_NET_SB1000=mCONFIG_PHYLINK=mCONFIG_PHYLIB=yCONFIG_SWPHY=yCONFIG_LED_TRIGGER_PHY=yCONFIG_FIXED_PHY=yCONFIG_SFP=m
#
# MII PHY device drivers
#CONFIG_AMD_PHY=mCONFIG_ADIN_PHY=m
# CONFIG_ADIN1100_PHY is not setCONFIG_AQUANTIA_PHY=mCONFIG_AX88796B_PHY=mCONFIG_BROADCOM_PHY=mCONFIG_BCM54140_PHY=mCONFIG_BCM7XXX_PHY=mCONFIG_BCM84881_PHY=yCONFIG_BCM87XX_PHY=mCONFIG_BCM_NET_PHYLIB=mCONFIG_BCM_NET_PHYPTP=mCONFIG_CICADA_PHY=mCONFIG_CORTINA_PHY=mCONFIG_DAVICOM_PHY=mCONFIG_ICPLUS_PHY=mCONFIG_LXT_PHY=mCONFIG_INTEL_XWAY_PHY=mCONFIG_LSI_ET1011C_PHY=mCONFIG_MARVELL_PHY=mCONFIG_MARVELL_10G_PHY=m
# CONFIG_MARVELL_88Q2XXX_PHY is not setCONFIG_MARVELL_88X2222_PHY=mCONFIG_MAXLINEAR_GPHY=mCONFIG_MEDIATEK_GE_PHY=m
# CONFIG_MEDIATEK_GE_SOC_PHY is not setCONFIG_MICREL_PHY=m
# CONFIG_MICROCHIP_T1S_PHY is not setCONFIG_MICROCHIP_PHY=mCONFIG_MICROCHIP_T1_PHY=mCONFIG_MICROSEMI_PHY=mCONFIG_MOTORCOMM_PHY=mCONFIG_NATIONAL_PHY=m
# CONFIG_NXP_CBTX_PHY is not setCONFIG_NXP_C45_TJA11XX_PHY=mCONFIG_NXP_TJA11XX_PHY=m
# CONFIG_NCN26000_PHY is not setCONFIG_QCOM_NET_PHYLIB=mCONFIG_AT803X_PHY=m
# CONFIG_QCA83XX_PHY is not set
# CONFIG_QCA808X_PHY is not setCONFIG_QSEMI_PHY=mCONFIG_REALTEK_PHY=mCONFIG_RENESAS_PHY=mCONFIG_ROCKCHIP_PHY=mCONFIG_SMSC_PHY=mCONFIG_STE10XP=mCONFIG_TERANETICS_PHY=mCONFIG_DP83822_PHY=mCONFIG_DP83TC811_PHY=mCONFIG_DP83848_PHY=mCONFIG_DP83867_PHY=mCONFIG_DP83869_PHY=m
# CONFIG_DP83TD510_PHY is not set
# CONFIG_DP83TG720_PHY is not setCONFIG_VITESSE_PHY=mCONFIG_XILINX_GMII2RGMII=mCONFIG_MICREL_KS8995MA=m
# CONFIG_PSE_CONTROLLER is not setCONFIG_CAN_DEV=mCONFIG_CAN_VCAN=mCONFIG_CAN_VXCAN=mCONFIG_CAN_NETLINK=yCONFIG_CAN_CALC_BITTIMING=yCONFIG_CAN_RX_OFFLOAD=y
# CONFIG_CAN_CAN327 is not setCONFIG_CAN_JANZ_ICAN3=mCONFIG_CAN_KVASER_PCIEFD=mCONFIG_CAN_SLCAN=mCONFIG_CAN_C_CAN=mCONFIG_CAN_C_CAN_PLATFORM=mCONFIG_CAN_C_CAN_PCI=mCONFIG_CAN_CC770=mCONFIG_CAN_CC770_ISA=mCONFIG_CAN_CC770_PLATFORM=m
# CONFIG_CAN_CTUCANFD_PCI is not set
# CONFIG_CAN_ESD_402_PCI is not setCONFIG_CAN_IFI_CANFD=mCONFIG_CAN_M_CAN=mCONFIG_CAN_M_CAN_PCI=mCONFIG_CAN_M_CAN_PLATFORM=mCONFIG_CAN_M_CAN_TCAN4X5X=mCONFIG_CAN_PEAK_PCIEFD=mCONFIG_CAN_SJA1000=mCONFIG_CAN_EMS_PCI=mCONFIG_CAN_EMS_PCMCIA=mCONFIG_CAN_F81601=mCONFIG_CAN_KVASER_PCI=mCONFIG_CAN_PEAK_PCI=mCONFIG_CAN_PEAK_PCIEC=yCONFIG_CAN_PEAK_PCMCIA=mCONFIG_CAN_PLX_PCI=mCONFIG_CAN_SJA1000_ISA=mCONFIG_CAN_SJA1000_PLATFORM=mCONFIG_CAN_SOFTING=mCONFIG_CAN_SOFTING_CS=m
#
# CAN SPI interfaces
#CONFIG_CAN_HI311X=mCONFIG_CAN_MCP251X=mCONFIG_CAN_MCP251XFD=m
# CONFIG_CAN_MCP251XFD_SANITY is not set
# end of CAN SPI interfaces
#
# CAN USB interfaces
#CONFIG_CAN_8DEV_USB=mCONFIG_CAN_EMS_USB=m
# CONFIG_CAN_ESD_USB is not setCONFIG_CAN_ETAS_ES58X=m
# CONFIG_CAN_F81604 is not setCONFIG_CAN_GS_USB=mCONFIG_CAN_KVASER_USB=mCONFIG_CAN_MCBA_USB=mCONFIG_CAN_PEAK_USB=mCONFIG_CAN_UCAN=m
# end of CAN USB interfaces
# CONFIG_CAN_DEBUG_DEVICES is not setCONFIG_MDIO_DEVICE=yCONFIG_MDIO_BUS=yCONFIG_FWNODE_MDIO=yCONFIG_ACPI_MDIO=yCONFIG_MDIO_DEVRES=yCONFIG_MDIO_BITBANG=mCONFIG_MDIO_BCM_UNIMAC=mCONFIG_MDIO_CAVIUM=mCONFIG_MDIO_GPIO=mCONFIG_MDIO_I2C=mCONFIG_MDIO_MVUSB=mCONFIG_MDIO_MSCC_MIIM=mCONFIG_MDIO_REGMAP=mCONFIG_MDIO_THUNDER=m
#
# MDIO Multiplexers
#
#
# PCS device drivers
#CONFIG_PCS_XPCS=mCONFIG_PCS_LYNX=mCONFIG_PCS_MTK_LYNXI=m
# end of PCS device drivers
CONFIG_PLIP=mCONFIG_PPP=yCONFIG_PPP_BSDCOMP=mCONFIG_PPP_DEFLATE=mCONFIG_PPP_FILTER=yCONFIG_PPP_MPPE=mCONFIG_PPP_MULTILINK=yCONFIG_PPPOATM=mCONFIG_PPPOE=m
# CONFIG_PPPOE_HASH_BITS_1 is not set
# CONFIG_PPPOE_HASH_BITS_2 is not setCONFIG_PPPOE_HASH_BITS_4=y
# CONFIG_PPPOE_HASH_BITS_8 is not setCONFIG_PPPOE_HASH_BITS=4CONFIG_PPTP=mCONFIG_PPPOL2TP=mCONFIG_PPP_ASYNC=mCONFIG_PPP_SYNC_TTY=mCONFIG_SLIP=mCONFIG_SLHC=yCONFIG_SLIP_COMPRESSED=yCONFIG_SLIP_SMART=yCONFIG_SLIP_MODE_SLIP6=yCONFIG_USB_NET_DRIVERS=mCONFIG_USB_CATC=mCONFIG_USB_KAWETH=mCONFIG_USB_PEGASUS=mCONFIG_USB_RTL8150=mCONFIG_USB_RTL8152=mCONFIG_USB_LAN78XX=mCONFIG_USB_USBNET=mCONFIG_USB_NET_AX8817X=mCONFIG_USB_NET_AX88179_178A=mCONFIG_USB_NET_CDCETHER=mCONFIG_USB_NET_CDC_EEM=mCONFIG_USB_NET_CDC_NCM=mCONFIG_USB_NET_HUAWEI_CDC_NCM=mCONFIG_USB_NET_CDC_MBIM=mCONFIG_USB_NET_DM9601=mCONFIG_USB_NET_SR9700=mCONFIG_USB_NET_SR9800=mCONFIG_USB_NET_SMSC75XX=mCONFIG_USB_NET_SMSC95XX=mCONFIG_USB_NET_GL620A=mCONFIG_USB_NET_NET1080=mCONFIG_USB_NET_PLUSB=mCONFIG_USB_NET_MCS7830=mCONFIG_USB_NET_RNDIS_HOST=mCONFIG_USB_NET_CDC_SUBSET_ENABLE=mCONFIG_USB_NET_CDC_SUBSET=mCONFIG_USB_ALI_M5632=yCONFIG_USB_AN2720=yCONFIG_USB_BELKIN=yCONFIG_USB_ARMLINUX=yCONFIG_USB_EPSON2888=yCONFIG_USB_KC2190=yCONFIG_USB_NET_ZAURUS=mCONFIG_USB_NET_CX82310_ETH=mCONFIG_USB_NET_KALMIA=mCONFIG_USB_NET_QMI_WWAN=mCONFIG_USB_HSO=mCONFIG_USB_NET_INT51X1=mCONFIG_USB_CDC_PHONET=mCONFIG_USB_IPHETH=mCONFIG_USB_SIERRA_NET=mCONFIG_USB_VL600=mCONFIG_USB_NET_CH9200=mCONFIG_USB_NET_AQC111=mCONFIG_USB_RTL8153_ECM=mCONFIG_WLAN=yCONFIG_WLAN_VENDOR_ADMTEK=yCONFIG_ADM8211=mCONFIG_ATH_COMMON=mCONFIG_WLAN_VENDOR_ATH=y
# CONFIG_ATH_DEBUG is not setCONFIG_ATH5K=m
# CONFIG_ATH5K_DEBUG is not set
# CONFIG_ATH5K_TRACER is not setCONFIG_ATH5K_PCI=yCONFIG_ATH9K_HW=mCONFIG_ATH9K_COMMON=mCONFIG_ATH9K_COMMON_DEBUG=yCONFIG_ATH9K_BTCOEX_SUPPORT=yCONFIG_ATH9K=mCONFIG_ATH9K_PCI=yCONFIG_ATH9K_AHB=yCONFIG_ATH9K_DEBUGFS=yCONFIG_ATH9K_STATION_STATISTICS=y
# CONFIG_ATH9K_DYNACK is not setCONFIG_ATH9K_WOW=yCONFIG_ATH9K_RFKILL=yCONFIG_ATH9K_CHANNEL_CONTEXT=yCONFIG_ATH9K_PCOEM=yCONFIG_ATH9K_PCI_NO_EEPROM=mCONFIG_ATH9K_HTC=mCONFIG_ATH9K_HTC_DEBUGFS=yCONFIG_ATH9K_HWRNG=yCONFIG_ATH9K_COMMON_SPECTRAL=yCONFIG_CARL9170=mCONFIG_CARL9170_LEDS=y
# CONFIG_CARL9170_DEBUGFS is not setCONFIG_CARL9170_WPC=yCONFIG_CARL9170_HWRNG=yCONFIG_ATH6KL=mCONFIG_ATH6KL_SDIO=mCONFIG_ATH6KL_USB=m
# CONFIG_ATH6KL_DEBUG is not set
# CONFIG_ATH6KL_TRACING is not setCONFIG_AR5523=mCONFIG_WIL6210=mCONFIG_WIL6210_ISR_COR=yCONFIG_WIL6210_TRACING=yCONFIG_WIL6210_DEBUGFS=yCONFIG_ATH10K=mCONFIG_ATH10K_CE=yCONFIG_ATH10K_PCI=mCONFIG_ATH10K_SDIO=mCONFIG_ATH10K_USB=m
# CONFIG_ATH10K_DEBUG is not setCONFIG_ATH10K_DEBUGFS=yCONFIG_ATH10K_SPECTRAL=yCONFIG_ATH10K_TRACING=yCONFIG_WCN36XX=m
# CONFIG_WCN36XX_DEBUGFS is not setCONFIG_ATH11K=mCONFIG_ATH11K_AHB=mCONFIG_ATH11K_PCI=m
# CONFIG_ATH11K_DEBUG is not setCONFIG_ATH11K_DEBUGFS=yCONFIG_ATH11K_TRACING=yCONFIG_ATH11K_SPECTRAL=y
# CONFIG_ATH12K is not setCONFIG_WLAN_VENDOR_ATMEL=yCONFIG_AT76C50X_USB=mCONFIG_WLAN_VENDOR_BROADCOM=yCONFIG_B43=mCONFIG_B43_BCMA=yCONFIG_B43_SSB=yCONFIG_B43_BUSES_BCMA_AND_SSB=y
# CONFIG_B43_BUSES_BCMA is not set
# CONFIG_B43_BUSES_SSB is not setCONFIG_B43_PCI_AUTOSELECT=yCONFIG_B43_PCICORE_AUTOSELECT=y
# CONFIG_B43_SDIO is not setCONFIG_B43_BCMA_PIO=yCONFIG_B43_PIO=yCONFIG_B43_PHY_G=yCONFIG_B43_PHY_N=yCONFIG_B43_PHY_LP=yCONFIG_B43_PHY_HT=yCONFIG_B43_LEDS=yCONFIG_B43_HWRNG=y
# CONFIG_B43_DEBUG is not setCONFIG_B43LEGACY=mCONFIG_B43LEGACY_PCI_AUTOSELECT=yCONFIG_B43LEGACY_PCICORE_AUTOSELECT=yCONFIG_B43LEGACY_LEDS=yCONFIG_B43LEGACY_HWRNG=y
# CONFIG_B43LEGACY_DEBUG is not setCONFIG_B43LEGACY_DMA=yCONFIG_B43LEGACY_PIO=yCONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
# CONFIG_B43LEGACY_DMA_MODE is not set
# CONFIG_B43LEGACY_PIO_MODE is not setCONFIG_BRCMUTIL=mCONFIG_BRCMSMAC=mCONFIG_BRCMSMAC_LEDS=yCONFIG_BRCMFMAC=mCONFIG_BRCMFMAC_PROTO_BCDC=yCONFIG_BRCMFMAC_PROTO_MSGBUF=yCONFIG_BRCMFMAC_SDIO=yCONFIG_BRCMFMAC_USB=yCONFIG_BRCMFMAC_PCIE=yCONFIG_BRCM_TRACING=y
# CONFIG_BRCMDBG is not setCONFIG_WLAN_VENDOR_INTEL=yCONFIG_IPW2100=mCONFIG_IPW2100_MONITOR=y
# CONFIG_IPW2100_DEBUG is not setCONFIG_IPW2200=mCONFIG_IPW2200_MONITOR=yCONFIG_IPW2200_RADIOTAP=yCONFIG_IPW2200_PROMISCUOUS=yCONFIG_IPW2200_QOS=y
# CONFIG_IPW2200_DEBUG is not setCONFIG_LIBIPW=m
# CONFIG_LIBIPW_DEBUG is not setCONFIG_IWLEGACY=mCONFIG_IWL4965=mCONFIG_IWL3945=m
#
# iwl3945 / iwl4965 Debugging Options # # CONFIG_IWLEGACY_DEBUG is not set CONFIG_IWLEGACY_DEBUGFS=y # end of iwl3945 / iwl4965 Debugging Options
CONFIG_IWLWIFI=mCONFIG_IWLWIFI_LEDS=yCONFIG_IWLDVM=mCONFIG_IWLMVM=mCONFIG_IWLWIFI_OPMODE_MODULAR=y
#
# Debugging Options
#
# CONFIG_IWLWIFI_DEBUG is not setCONFIG_IWLWIFI_DEBUGFS=yCONFIG_IWLWIFI_DEVICE_TRACING=y
# end of Debugging Options
CONFIG_WLAN_VENDOR_INTERSIL=yCONFIG_P54_COMMON=mCONFIG_P54_USB=mCONFIG_P54_PCI=mCONFIG_P54_SPI=m
# CONFIG_P54_SPI_DEFAULT_EEPROM is not setCONFIG_P54_LEDS=yCONFIG_WLAN_VENDOR_MARVELL=yCONFIG_LIBERTAS=mCONFIG_LIBERTAS_USB=mCONFIG_LIBERTAS_SDIO=mCONFIG_LIBERTAS_SPI=m
# CONFIG_LIBERTAS_DEBUG is not setCONFIG_LIBERTAS_MESH=yCONFIG_LIBERTAS_THINFIRM=m
# CONFIG_LIBERTAS_THINFIRM_DEBUG is not setCONFIG_LIBERTAS_THINFIRM_USB=mCONFIG_MWIFIEX=mCONFIG_MWIFIEX_SDIO=mCONFIG_MWIFIEX_PCIE=mCONFIG_MWIFIEX_USB=mCONFIG_MWL8K=mCONFIG_WLAN_VENDOR_MEDIATEK=yCONFIG_MT7601U=mCONFIG_MT76_CORE=mCONFIG_MT76_LEDS=yCONFIG_MT76_USB=mCONFIG_MT76_SDIO=m
CONFIG_MT76x02_LIB=m
CONFIG_MT76x02_USB=mCONFIG_MT76_CONNAC_LIB=m
CONFIG_MT792x_LIB=m
CONFIG_MT76x0_COMMON=m
CONFIG_MT76x0U=m
CONFIG_MT76x0E=m
CONFIG_MT76x2_COMMON=m
CONFIG_MT76x2E=m
CONFIG_MT76x2U=mCONFIG_MT7603E=mCONFIG_MT7615_COMMON=mCONFIG_MT7615E=mCONFIG_MT7663_USB_SDIO_COMMON=mCONFIG_MT7663U=mCONFIG_MT7663S=mCONFIG_MT7915E=mCONFIG_MT7921_COMMON=mCONFIG_MT7921E=m
# CONFIG_MT7921S is not set
# CONFIG_MT7921U is not set
# CONFIG_MT7996E is not set
# CONFIG_MT7925E is not set
# CONFIG_MT7925U is not setCONFIG_WLAN_VENDOR_MICROCHIP=yCONFIG_WILC1000=mCONFIG_WILC1000_SDIO=mCONFIG_WILC1000_SPI=mCONFIG_WILC1000_HW_OOB_INTR=yCONFIG_WLAN_VENDOR_PURELIFI=y
# CONFIG_PLFXLC is not setCONFIG_WLAN_VENDOR_RALINK=yCONFIG_RT2X00=mCONFIG_RT2400PCI=mCONFIG_RT2500PCI=mCONFIG_RT61PCI=mCONFIG_RT2800PCI=mCONFIG_RT2800PCI_RT33XX=yCONFIG_RT2800PCI_RT35XX=yCONFIG_RT2800PCI_RT53XX=yCONFIG_RT2800PCI_RT3290=yCONFIG_RT2500USB=mCONFIG_RT73USB=mCONFIG_RT2800USB=mCONFIG_RT2800USB_RT33XX=yCONFIG_RT2800USB_RT35XX=yCONFIG_RT2800USB_RT3573=yCONFIG_RT2800USB_RT53XX=yCONFIG_RT2800USB_RT55XX=yCONFIG_RT2800USB_UNKNOWN=yCONFIG_RT2800_LIB=mCONFIG_RT2800_LIB_MMIO=mCONFIG_RT2X00_LIB_MMIO=mCONFIG_RT2X00_LIB_PCI=mCONFIG_RT2X00_LIB_USB=mCONFIG_RT2X00_LIB=mCONFIG_RT2X00_LIB_FIRMWARE=yCONFIG_RT2X00_LIB_CRYPTO=yCONFIG_RT2X00_LIB_LEDS=y
# CONFIG_RT2X00_LIB_DEBUGFS is not set
# CONFIG_RT2X00_DEBUG is not setCONFIG_WLAN_VENDOR_REALTEK=yCONFIG_RTL8180=mCONFIG_RTL8187=mCONFIG_RTL8187_LEDS=yCONFIG_RTL_CARDS=mCONFIG_RTL8192CE=mCONFIG_RTL8192SE=mCONFIG_RTL8192DE=mCONFIG_RTL8723AE=mCONFIG_RTL8723BE=mCONFIG_RTL8188EE=mCONFIG_RTL8192EE=mCONFIG_RTL8821AE=mCONFIG_RTL8192CU=mCONFIG_RTLWIFI=mCONFIG_RTLWIFI_PCI=mCONFIG_RTLWIFI_USB=m
# CONFIG_RTLWIFI_DEBUG is not setCONFIG_RTL8192C_COMMON=mCONFIG_RTL8723_COMMON=mCONFIG_RTLBTCOEXIST=mCONFIG_RTL8XXXU=mCONFIG_RTL8XXXU_UNTESTED=yCONFIG_RTW88=mCONFIG_RTW88_CORE=mCONFIG_RTW88_PCI=mCONFIG_RTW88_8822B=mCONFIG_RTW88_8822C=mCONFIG_RTW88_8723D=mCONFIG_RTW88_8821C=mCONFIG_RTW88_8822BE=m
# CONFIG_RTW88_8822BS is not set
# CONFIG_RTW88_8822BU is not setCONFIG_RTW88_8822CE=m
# CONFIG_RTW88_8822CS is not set
# CONFIG_RTW88_8822CU is not setCONFIG_RTW88_8723DE=m
# CONFIG_RTW88_8723DS is not set
# CONFIG_RTW88_8723DU is not setCONFIG_RTW88_8821CE=m
# CONFIG_RTW88_8821CS is not set
# CONFIG_RTW88_8821CU is not setCONFIG_RTW88_DEBUG=yCONFIG_RTW88_DEBUGFS=yCONFIG_RTW89=mCONFIG_RTW89_CORE=mCONFIG_RTW89_PCI=mCONFIG_RTW89_8852A=m
# CONFIG_RTW89_8851BE is not setCONFIG_RTW89_8852AE=m
# CONFIG_RTW89_8852BE is not set
# CONFIG_RTW89_8852CE is not setCONFIG_RTW89_DEBUG=yCONFIG_RTW89_DEBUGMSG=yCONFIG_RTW89_DEBUGFS=yCONFIG_WLAN_VENDOR_RSI=yCONFIG_RSI_91X=m
# CONFIG_RSI_DEBUGFS is not setCONFIG_RSI_SDIO=mCONFIG_RSI_USB=mCONFIG_RSI_COEX=yCONFIG_WLAN_VENDOR_SILABS=yCONFIG_WFX=mCONFIG_WLAN_VENDOR_ST=yCONFIG_CW1200=mCONFIG_CW1200_WLAN_SDIO=mCONFIG_CW1200_WLAN_SPI=mCONFIG_WLAN_VENDOR_TI=yCONFIG_WL1251=mCONFIG_WL1251_SPI=mCONFIG_WL1251_SDIO=mCONFIG_WL12XX=mCONFIG_WL18XX=mCONFIG_WLCORE=mCONFIG_WLCORE_SDIO=mCONFIG_WLAN_VENDOR_ZYDAS=yCONFIG_ZD1211RW=m
# CONFIG_ZD1211RW_DEBUG is not setCONFIG_WLAN_VENDOR_QUANTENNA=yCONFIG_QTNFMAC=mCONFIG_QTNFMAC_PCIE=mCONFIG_MAC80211_HWSIM=mCONFIG_VIRT_WIFI=mCONFIG_WAN=yCONFIG_HDLC=mCONFIG_HDLC_RAW=mCONFIG_HDLC_RAW_ETH=mCONFIG_HDLC_CISCO=mCONFIG_HDLC_FR=mCONFIG_HDLC_PPP=mCONFIG_HDLC_X25=m
# CONFIG_FRAMER is not setCONFIG_PCI200SYN=mCONFIG_WANXL=mCONFIG_PC300TOO=mCONFIG_FARSYNC=mCONFIG_LAPBETHER=mCONFIG_IEEE802154_DRIVERS=mCONFIG_IEEE802154_FAKELB=mCONFIG_IEEE802154_AT86RF230=mCONFIG_IEEE802154_MRF24J40=mCONFIG_IEEE802154_CC2520=mCONFIG_IEEE802154_ATUSB=mCONFIG_IEEE802154_ADF7242=mCONFIG_IEEE802154_CA8210=mCONFIG_IEEE802154_CA8210_DEBUGFS=yCONFIG_IEEE802154_MCR20A=mCONFIG_IEEE802154_HWSIM=m
#
# Wireless WAN
#CONFIG_WWAN=yCONFIG_WWAN_DEBUGFS=yCONFIG_WWAN_HWSIM=mCONFIG_MHI_WWAN_CTRL=mCONFIG_MHI_WWAN_MBIM=mCONFIG_RPMSG_WWAN_CTRL=mCONFIG_IOSM=m
# CONFIG_MTK_T7XX is not set
# end of Wireless WAN
CONFIG_XEN_NETDEV_FRONTEND=yCONFIG_XEN_NETDEV_BACKEND=mCONFIG_VMXNET3=mCONFIG_FUJITSU_ES=mCONFIG_USB4_NET=mCONFIG_HYPERV_NET=mCONFIG_NETDEVSIM=mCONFIG_NET_FAILOVER=mCONFIG_ISDN=yCONFIG_ISDN_CAPI=yCONFIG_CAPI_TRACE=yCONFIG_ISDN_CAPI_MIDDLEWARE=yCONFIG_MISDN=mCONFIG_MISDN_DSP=mCONFIG_MISDN_L1OIP=m
#
# mISDN hardware drivers
#CONFIG_MISDN_HFCPCI=mCONFIG_MISDN_HFCMULTI=mCONFIG_MISDN_HFCUSB=mCONFIG_MISDN_AVMFRITZ=mCONFIG_MISDN_SPEEDFAX=mCONFIG_MISDN_INFINEON=mCONFIG_MISDN_W6692=mCONFIG_MISDN_NETJET=mCONFIG_MISDN_HDLC=mCONFIG_MISDN_IPAC=mCONFIG_MISDN_ISAR=m
#
# Input device support
#CONFIG_INPUT=yCONFIG_INPUT_LEDS=mCONFIG_INPUT_FF_MEMLESS=mCONFIG_INPUT_SPARSEKMAP=mCONFIG_INPUT_MATRIXKMAP=mCONFIG_INPUT_VIVALDIFMAP=y
#
# Userland interfaces
#CONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_PSAUX=yCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768CONFIG_INPUT_JOYDEV=mCONFIG_INPUT_EVDEV=yCONFIG_INPUT_EVBUG=m
#
# Input Device Drivers
#CONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ADC=mCONFIG_KEYBOARD_ADP5520=mCONFIG_KEYBOARD_ADP5588=mCONFIG_KEYBOARD_ADP5589=mCONFIG_KEYBOARD_APPLESPI=mCONFIG_KEYBOARD_ATKBD=yCONFIG_KEYBOARD_QT1050=mCONFIG_KEYBOARD_QT1070=mCONFIG_KEYBOARD_QT2160=mCONFIG_KEYBOARD_DLINK_DIR685=mCONFIG_KEYBOARD_LKKBD=mCONFIG_KEYBOARD_GPIO=mCONFIG_KEYBOARD_GPIO_POLLED=mCONFIG_KEYBOARD_TCA6416=mCONFIG_KEYBOARD_TCA8418=mCONFIG_KEYBOARD_MATRIX=mCONFIG_KEYBOARD_LM8323=mCONFIG_KEYBOARD_LM8333=mCONFIG_KEYBOARD_MAX7359=mCONFIG_KEYBOARD_MCS=mCONFIG_KEYBOARD_MPR121=mCONFIG_KEYBOARD_NEWTON=mCONFIG_KEYBOARD_OPENCORES=m
# CONFIG_KEYBOARD_PINEPHONE is not setCONFIG_KEYBOARD_SAMSUNG=mCONFIG_KEYBOARD_STOWAWAY=mCONFIG_KEYBOARD_SUNKBD=mCONFIG_KEYBOARD_IQS62X=mCONFIG_KEYBOARD_TM2_TOUCHKEY=mCONFIG_KEYBOARD_TWL4030=mCONFIG_KEYBOARD_XTKBD=mCONFIG_KEYBOARD_CROS_EC=mCONFIG_KEYBOARD_MTK_PMIC=m
# CONFIG_KEYBOARD_CYPRESS_SF is not setCONFIG_INPUT_MOUSE=yCONFIG_MOUSE_PS2=mCONFIG_MOUSE_PS2_ALPS=yCONFIG_MOUSE_PS2_BYD=yCONFIG_MOUSE_PS2_LOGIPS2PP=yCONFIG_MOUSE_PS2_SYNAPTICS=yCONFIG_MOUSE_PS2_SYNAPTICS_SMBUS=yCONFIG_MOUSE_PS2_CYPRESS=yCONFIG_MOUSE_PS2_LIFEBOOK=yCONFIG_MOUSE_PS2_TRACKPOINT=yCONFIG_MOUSE_PS2_ELANTECH=yCONFIG_MOUSE_PS2_ELANTECH_SMBUS=yCONFIG_MOUSE_PS2_SENTELIC=yCONFIG_MOUSE_PS2_TOUCHKIT=yCONFIG_MOUSE_PS2_FOCALTECH=yCONFIG_MOUSE_PS2_VMMOUSE=yCONFIG_MOUSE_PS2_SMBUS=yCONFIG_MOUSE_SERIAL=mCONFIG_MOUSE_APPLETOUCH=mCONFIG_MOUSE_BCM5974=mCONFIG_MOUSE_CYAPA=mCONFIG_MOUSE_ELAN_I2C=mCONFIG_MOUSE_ELAN_I2C_I2C=yCONFIG_MOUSE_ELAN_I2C_SMBUS=yCONFIG_MOUSE_VSXXXAA=mCONFIG_MOUSE_GPIO=mCONFIG_MOUSE_SYNAPTICS_I2C=mCONFIG_MOUSE_SYNAPTICS_USB=mCONFIG_INPUT_JOYSTICK=yCONFIG_JOYSTICK_ANALOG=mCONFIG_JOYSTICK_A3D=mCONFIG_JOYSTICK_ADC=mCONFIG_JOYSTICK_ADI=mCONFIG_JOYSTICK_COBRA=mCONFIG_JOYSTICK_GF2K=mCONFIG_JOYSTICK_GRIP=mCONFIG_JOYSTICK_GRIP_MP=mCONFIG_JOYSTICK_GUILLEMOT=mCONFIG_JOYSTICK_INTERACT=mCONFIG_JOYSTICK_SIDEWINDER=mCONFIG_JOYSTICK_TMDC=mCONFIG_JOYSTICK_IFORCE=mCONFIG_JOYSTICK_IFORCE_USB=mCONFIG_JOYSTICK_IFORCE_232=mCONFIG_JOYSTICK_WARRIOR=mCONFIG_JOYSTICK_MAGELLAN=mCONFIG_JOYSTICK_SPACEORB=mCONFIG_JOYSTICK_SPACEBALL=mCONFIG_JOYSTICK_STINGER=mCONFIG_JOYSTICK_TWIDJOY=mCONFIG_JOYSTICK_ZHENHUA=mCONFIG_JOYSTICK_DB9=mCONFIG_JOYSTICK_GAMECON=mCONFIG_JOYSTICK_TURBOGRAFX=mCONFIG_JOYSTICK_AS5011=mCONFIG_JOYSTICK_JOYDUMP=mCONFIG_JOYSTICK_XPAD=mCONFIG_JOYSTICK_XPAD_FF=yCONFIG_JOYSTICK_XPAD_LEDS=yCONFIG_JOYSTICK_WALKERA0701=mCONFIG_JOYSTICK_PSXPAD_SPI=mCONFIG_JOYSTICK_PSXPAD_SPI_FF=yCONFIG_JOYSTICK_PXRC=mCONFIG_JOYSTICK_QWIIC=mCONFIG_JOYSTICK_FSIA6B=m
# CONFIG_JOYSTICK_SENSEHAT is not set
# CONFIG_JOYSTICK_SEESAW is not setCONFIG_INPUT_TABLET=yCONFIG_TABLET_USB_ACECAD=mCONFIG_TABLET_USB_AIPTEK=mCONFIG_TABLET_USB_HANWANG=mCONFIG_TABLET_USB_KBTAB=mCONFIG_TABLET_USB_PEGASUS=mCONFIG_TABLET_SERIAL_WACOM4=mCONFIG_INPUT_TOUCHSCREEN=yCONFIG_TOUCHSCREEN_88PM860X=mCONFIG_TOUCHSCREEN_ADS7846=mCONFIG_TOUCHSCREEN_AD7877=mCONFIG_TOUCHSCREEN_AD7879=mCONFIG_TOUCHSCREEN_AD7879_I2C=mCONFIG_TOUCHSCREEN_AD7879_SPI=mCONFIG_TOUCHSCREEN_ADC=mCONFIG_TOUCHSCREEN_ATMEL_MXT=mCONFIG_TOUCHSCREEN_ATMEL_MXT_T37=yCONFIG_TOUCHSCREEN_AUO_PIXCIR=mCONFIG_TOUCHSCREEN_BU21013=mCONFIG_TOUCHSCREEN_BU21029=mCONFIG_TOUCHSCREEN_CHIPONE_ICN8505=mCONFIG_TOUCHSCREEN_CY8CTMA140=mCONFIG_TOUCHSCREEN_CY8CTMG110=mCONFIG_TOUCHSCREEN_CYTTSP_CORE=mCONFIG_TOUCHSCREEN_CYTTSP_I2C=mCONFIG_TOUCHSCREEN_CYTTSP_SPI=mCONFIG_TOUCHSCREEN_CYTTSP4_CORE=mCONFIG_TOUCHSCREEN_CYTTSP4_I2C=mCONFIG_TOUCHSCREEN_CYTTSP4_SPI=m
# CONFIG_TOUCHSCREEN_CYTTSP5 is not setCONFIG_TOUCHSCREEN_DA9034=mCONFIG_TOUCHSCREEN_DA9052=mCONFIG_TOUCHSCREEN_DYNAPRO=mCONFIG_TOUCHSCREEN_HAMPSHIRE=mCONFIG_TOUCHSCREEN_EETI=mCONFIG_TOUCHSCREEN_EGALAX_SERIAL=mCONFIG_TOUCHSCREEN_EXC3000=mCONFIG_TOUCHSCREEN_FUJITSU=mCONFIG_TOUCHSCREEN_GOODIX=m
# CONFIG_TOUCHSCREEN_GOODIX_BERLIN_I2C is not set
# CONFIG_TOUCHSCREEN_GOODIX_BERLIN_SPI is not setCONFIG_TOUCHSCREEN_HIDEEP=mCONFIG_TOUCHSCREEN_HYCON_HY46XX=m
# CONFIG_TOUCHSCREEN_HYNITRON_CSTXXX is not setCONFIG_TOUCHSCREEN_ILI210X=mCONFIG_TOUCHSCREEN_ILITEK=mCONFIG_TOUCHSCREEN_S6SY761=mCONFIG_TOUCHSCREEN_GUNZE=mCONFIG_TOUCHSCREEN_EKTF2127=mCONFIG_TOUCHSCREEN_ELAN=yCONFIG_TOUCHSCREEN_ELO=mCONFIG_TOUCHSCREEN_WACOM_W8001=mCONFIG_TOUCHSCREEN_WACOM_I2C=mCONFIG_TOUCHSCREEN_MAX11801=mCONFIG_TOUCHSCREEN_MCS5000=mCONFIG_TOUCHSCREEN_MMS114=mCONFIG_TOUCHSCREEN_MELFAS_MIP4=mCONFIG_TOUCHSCREEN_MSG2638=mCONFIG_TOUCHSCREEN_MTOUCH=m
# CONFIG_TOUCHSCREEN_NOVATEK_NVT_TS is not set
# CONFIG_TOUCHSCREEN_IMAGIS is not setCONFIG_TOUCHSCREEN_INEXIO=mCONFIG_TOUCHSCREEN_PENMOUNT=mCONFIG_TOUCHSCREEN_EDT_FT5X06=mCONFIG_TOUCHSCREEN_TOUCHRIGHT=mCONFIG_TOUCHSCREEN_TOUCHWIN=mCONFIG_TOUCHSCREEN_PIXCIR=mCONFIG_TOUCHSCREEN_WDT87XX_I2C=mCONFIG_TOUCHSCREEN_WM831X=mCONFIG_TOUCHSCREEN_WM97XX=mCONFIG_TOUCHSCREEN_WM9705=yCONFIG_TOUCHSCREEN_WM9712=yCONFIG_TOUCHSCREEN_WM9713=yCONFIG_TOUCHSCREEN_USB_COMPOSITE=mCONFIG_TOUCHSCREEN_MC13783=mCONFIG_TOUCHSCREEN_USB_EGALAX=yCONFIG_TOUCHSCREEN_USB_PANJIT=yCONFIG_TOUCHSCREEN_USB_3M=yCONFIG_TOUCHSCREEN_USB_ITM=yCONFIG_TOUCHSCREEN_USB_ETURBO=yCONFIG_TOUCHSCREEN_USB_GUNZE=yCONFIG_TOUCHSCREEN_USB_DMC_TSC10=yCONFIG_TOUCHSCREEN_USB_IRTOUCH=yCONFIG_TOUCHSCREEN_USB_IDEALTEK=yCONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=yCONFIG_TOUCHSCREEN_USB_GOTOP=yCONFIG_TOUCHSCREEN_USB_JASTEC=yCONFIG_TOUCHSCREEN_USB_ELO=yCONFIG_TOUCHSCREEN_USB_E2I=yCONFIG_TOUCHSCREEN_USB_ZYTRONIC=yCONFIG_TOUCHSCREEN_USB_ETT_TC45USB=yCONFIG_TOUCHSCREEN_USB_NEXIO=yCONFIG_TOUCHSCREEN_USB_EASYTOUCH=yCONFIG_TOUCHSCREEN_TOUCHIT213=mCONFIG_TOUCHSCREEN_TSC_SERIO=mCONFIG_TOUCHSCREEN_TSC200X_CORE=mCONFIG_TOUCHSCREEN_TSC2004=mCONFIG_TOUCHSCREEN_TSC2005=mCONFIG_TOUCHSCREEN_TSC2007=mCONFIG_TOUCHSCREEN_TSC2007_IIO=yCONFIG_TOUCHSCREEN_PCAP=mCONFIG_TOUCHSCREEN_RM_TS=mCONFIG_TOUCHSCREEN_SILEAD=mCONFIG_TOUCHSCREEN_SIS_I2C=mCONFIG_TOUCHSCREEN_ST1232=mCONFIG_TOUCHSCREEN_STMFTS=mCONFIG_TOUCHSCREEN_SUR40=mCONFIG_TOUCHSCREEN_SURFACE3_SPI=mCONFIG_TOUCHSCREEN_SX8654=mCONFIG_TOUCHSCREEN_TPS6507X=mCONFIG_TOUCHSCREEN_ZET6223=mCONFIG_TOUCHSCREEN_ZFORCE=m
# CONFIG_TOUCHSCREEN_COLIBRI_VF50 is not setCONFIG_TOUCHSCREEN_ROHM_BU21023=mCONFIG_TOUCHSCREEN_IQS5XX=m
# CONFIG_TOUCHSCREEN_IQS7211 is not setCONFIG_TOUCHSCREEN_ZINITIX=m
# CONFIG_TOUCHSCREEN_HIMAX_HX83112B is not setCONFIG_INPUT_MISC=yCONFIG_INPUT_88PM860X_ONKEY=mCONFIG_INPUT_88PM80X_ONKEY=mCONFIG_INPUT_AD714X=mCONFIG_INPUT_AD714X_I2C=mCONFIG_INPUT_AD714X_SPI=mCONFIG_INPUT_ARIZONA_HAPTICS=mCONFIG_INPUT_ATC260X_ONKEY=mCONFIG_INPUT_BMA150=mCONFIG_INPUT_E3X0_BUTTON=mCONFIG_INPUT_PCSPKR=mCONFIG_INPUT_MAX77693_HAPTIC=mCONFIG_INPUT_MAX8925_ONKEY=mCONFIG_INPUT_MAX8997_HAPTIC=mCONFIG_INPUT_MC13783_PWRBUTTON=mCONFIG_INPUT_MMA8450=mCONFIG_INPUT_APANEL=mCONFIG_INPUT_GPIO_BEEPER=mCONFIG_INPUT_GPIO_DECODER=mCONFIG_INPUT_GPIO_VIBRA=mCONFIG_INPUT_ATLAS_BTNS=mCONFIG_INPUT_ATI_REMOTE2=mCONFIG_INPUT_KEYSPAN_REMOTE=mCONFIG_INPUT_KXTJ9=mCONFIG_INPUT_POWERMATE=mCONFIG_INPUT_YEALINK=mCONFIG_INPUT_CM109=mCONFIG_INPUT_REGULATOR_HAPTIC=mCONFIG_INPUT_RETU_PWRBUTTON=mCONFIG_INPUT_AXP20X_PEK=mCONFIG_INPUT_TWL4030_PWRBUTTON=mCONFIG_INPUT_TWL4030_VIBRA=mCONFIG_INPUT_TWL6040_VIBRA=mCONFIG_INPUT_UINPUT=yCONFIG_INPUT_PALMAS_PWRBUTTON=mCONFIG_INPUT_PCF50633_PMU=mCONFIG_INPUT_PCF8574=mCONFIG_INPUT_PWM_BEEPER=mCONFIG_INPUT_PWM_VIBRA=mCONFIG_INPUT_GPIO_ROTARY_ENCODER=mCONFIG_INPUT_DA7280_HAPTICS=mCONFIG_INPUT_DA9052_ONKEY=mCONFIG_INPUT_DA9055_ONKEY=mCONFIG_INPUT_DA9063_ONKEY=mCONFIG_INPUT_WM831X_ON=mCONFIG_INPUT_PCAP=mCONFIG_INPUT_ADXL34X=mCONFIG_INPUT_ADXL34X_I2C=mCONFIG_INPUT_ADXL34X_SPI=mCONFIG_INPUT_IMS_PCU=mCONFIG_INPUT_IQS269A=mCONFIG_INPUT_IQS626A=m
# CONFIG_INPUT_IQS7222 is not setCONFIG_INPUT_CMA3000=mCONFIG_INPUT_CMA3000_I2C=mCONFIG_INPUT_XEN_KBDDEV_FRONTEND=mCONFIG_INPUT_IDEAPAD_SLIDEBAR=mCONFIG_INPUT_SOC_BUTTON_ARRAY=mCONFIG_INPUT_DRV260X_HAPTICS=mCONFIG_INPUT_DRV2665_HAPTICS=mCONFIG_INPUT_DRV2667_HAPTICS=mCONFIG_INPUT_RAVE_SP_PWRBUTTON=mCONFIG_RMI4_CORE=mCONFIG_RMI4_I2C=mCONFIG_RMI4_SPI=mCONFIG_RMI4_SMB=mCONFIG_RMI4_F03=yCONFIG_RMI4_F03_SERIO=mCONFIG_RMI4_2D_SENSOR=yCONFIG_RMI4_F11=yCONFIG_RMI4_F12=yCONFIG_RMI4_F30=yCONFIG_RMI4_F34=yCONFIG_RMI4_F3A=yCONFIG_RMI4_F54=yCONFIG_RMI4_F55=y
#
# Hardware I/O ports
#CONFIG_SERIO=yCONFIG_ARCH_MIGHT_HAVE_PC_SERIO=yCONFIG_SERIO_I8042=yCONFIG_SERIO_SERPORT=mCONFIG_SERIO_CT82C710=mCONFIG_SERIO_PARKBD=mCONFIG_SERIO_PCIPS2=mCONFIG_SERIO_LIBPS2=yCONFIG_SERIO_RAW=mCONFIG_SERIO_ALTERA_PS2=mCONFIG_SERIO_PS2MULT=mCONFIG_SERIO_ARC_PS2=mCONFIG_HYPERV_KEYBOARD=mCONFIG_SERIO_GPIO_PS2=mCONFIG_USERIO=mCONFIG_GAMEPORT=mCONFIG_GAMEPORT_EMU10K1=mCONFIG_GAMEPORT_FM801=m
# end of Hardware I/O ports # end of Input device support
#
# Character devices
#CONFIG_TTY=yCONFIG_VT=yCONFIG_CONSOLE_TRANSLATIONS=yCONFIG_VT_CONSOLE=yCONFIG_VT_CONSOLE_SLEEP=yCONFIG_VT_HW_CONSOLE_BINDING=yCONFIG_UNIX98_PTYS=yCONFIG_LEGACY_PTYS=yCONFIG_LEGACY_PTY_COUNT=0CONFIG_LEGACY_TIOCSTI=yCONFIG_LDISC_AUTOLOAD=y
#
# Serial drivers
#CONFIG_SERIAL_EARLYCON=y CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set CONFIG_SERIAL_8250_PNP=y CONFIG_SERIAL_8250_16550A_VARIANTS=y CONFIG_SERIAL_8250_FINTEK=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_DMA=y CONFIG_SERIAL_8250_PCILIB=y CONFIG_SERIAL_8250_PCI=y CONFIG_SERIAL_8250_EXAR=m CONFIG_SERIAL_8250_CS=m CONFIG_SERIAL_8250_MEN_MCB=m CONFIG_SERIAL_8250_NR_UARTS=48 CONFIG_SERIAL_8250_RUNTIME_UARTS=32 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y
# CONFIG_SERIAL_8250_PCI1XXXX is not set CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set CONFIG_SERIAL_8250_RSA=y CONFIG_SERIAL_8250_DWLIB=y
# CONFIG_SERIAL_8250_DFL is not set CONFIG_SERIAL_8250_DW=m CONFIG_SERIAL_8250_RT288X=y CONFIG_SERIAL_8250_LPSS=m CONFIG_SERIAL_8250_MID=y CONFIG_SERIAL_8250_PERICOM=y
#
# Non-8250 serial port support
#CONFIG_SERIAL_KGDB_NMI=yCONFIG_SERIAL_MAX3100=mCONFIG_SERIAL_MAX310X=yCONFIG_SERIAL_UARTLITE=mCONFIG_SERIAL_UARTLITE_NR_UARTS=1CONFIG_SERIAL_CORE=yCONFIG_SERIAL_CORE_CONSOLE=yCONFIG_CONSOLE_POLL=yCONFIG_SERIAL_JSM=mCONFIG_SERIAL_LANTIQ=mCONFIG_SERIAL_SCCNXP=yCONFIG_SERIAL_SCCNXP_CONSOLE=yCONFIG_SERIAL_SC16IS7XX_CORE=mCONFIG_SERIAL_SC16IS7XX=mCONFIG_SERIAL_SC16IS7XX_I2C=yCONFIG_SERIAL_SC16IS7XX_SPI=yCONFIG_SERIAL_ALTERA_JTAGUART=mCONFIG_SERIAL_ALTERA_UART=mCONFIG_SERIAL_ALTERA_UART_MAXPORTS=4CONFIG_SERIAL_ALTERA_UART_BAUDRATE=115200CONFIG_SERIAL_ARC=mCONFIG_SERIAL_ARC_NR_PORTS=1CONFIG_SERIAL_RP2=mCONFIG_SERIAL_RP2_NR_UARTS=32CONFIG_SERIAL_FSL_LPUART=mCONFIG_SERIAL_FSL_LINFLEXUART=mCONFIG_SERIAL_MEN_Z135=mCONFIG_SERIAL_SPRD=m
# end of Serial drivers
CONFIG_SERIAL_MCTRL_GPIO=yCONFIG_SERIAL_NONSTANDARD=yCONFIG_MOXA_INTELLIO=mCONFIG_MOXA_SMARTIO=mCONFIG_N_HDLC=mCONFIG_IPWIRELESS=mCONFIG_N_GSM=mCONFIG_NOZOMI=mCONFIG_NULL_TTY=mCONFIG_HVC_DRIVER=yCONFIG_HVC_IRQ=yCONFIG_HVC_XEN=yCONFIG_HVC_XEN_FRONTEND=y
# CONFIG_RPMSG_TTY is not setCONFIG_SERIAL_DEV_BUS=yCONFIG_SERIAL_DEV_CTRL_TTYPORT=yCONFIG_TTY_PRINTK=yCONFIG_TTY_PRINTK_LEVEL=6CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not setCONFIG_PPDEV=mCONFIG_VIRTIO_CONSOLE=yCONFIG_IPMI_HANDLER=mCONFIG_IPMI_DMI_DECODE=yCONFIG_IPMI_PLAT_DATA=y
# CONFIG_IPMI_PANIC_EVENT is not setCONFIG_IPMI_DEVICE_INTERFACE=mCONFIG_IPMI_SI=mCONFIG_IPMI_SSIF=mCONFIG_IPMI_WATCHDOG=mCONFIG_IPMI_POWEROFF=mCONFIG_HW_RANDOM=yCONFIG_HW_RANDOM_TIMERIOMEM=mCONFIG_HW_RANDOM_INTEL=mCONFIG_HW_RANDOM_AMD=mCONFIG_HW_RANDOM_BA431=mCONFIG_HW_RANDOM_VIA=mCONFIG_HW_RANDOM_VIRTIO=mCONFIG_HW_RANDOM_XIPHERA=mCONFIG_APPLICOM=mCONFIG_MWAVE=mCONFIG_DEVMEM=yCONFIG_NVRAM=mCONFIG_DEVPORT=yCONFIG_HPET=yCONFIG_HPET_MMAP=yCONFIG_HPET_MMAP_DEFAULT=yCONFIG_HANGCHECK_TIMER=mCONFIG_UV_MMTIMER=mCONFIG_TCG_TPM=yCONFIG_HW_RANDOM_TPM=yCONFIG_TCG_TIS_CORE=yCONFIG_TCG_TIS=yCONFIG_TCG_TIS_SPI=mCONFIG_TCG_TIS_SPI_CR50=y
# CONFIG_TCG_TIS_I2C is not setCONFIG_TCG_TIS_I2C_CR50=mCONFIG_TCG_TIS_I2C_ATMEL=mCONFIG_TCG_TIS_I2C_INFINEON=mCONFIG_TCG_TIS_I2C_NUVOTON=mCONFIG_TCG_NSC=mCONFIG_TCG_ATMEL=mCONFIG_TCG_INFINEON=mCONFIG_TCG_XEN=mCONFIG_TCG_CRB=yCONFIG_TCG_VTPM_PROXY=mCONFIG_TCG_TIS_ST33ZP24=mCONFIG_TCG_TIS_ST33ZP24_I2C=mCONFIG_TCG_TIS_ST33ZP24_SPI=mCONFIG_TELCLOCK=mCONFIG_XILLYBUS_CLASS=mCONFIG_XILLYBUS=mCONFIG_XILLYBUS_PCIE=mCONFIG_XILLYUSB=m
# end of Character devices
#
# I2C support
#CONFIG_I2C=yCONFIG_ACPI_I2C_OPREGION=yCONFIG_I2C_BOARDINFO=yCONFIG_I2C_COMPAT=yCONFIG_I2C_CHARDEV=yCONFIG_I2C_MUX=m
#
# Multiplexer I2C Chip support
#CONFIG_I2C_MUX_GPIO=mCONFIG_I2C_MUX_LTC4306=mCONFIG_I2C_MUX_PCA9541=m
CONFIG_I2C_MUX_PCA954x=mCONFIG_I2C_MUX_REG=mCONFIG_I2C_MUX_MLXCPLD=m
# end of Multiplexer I2C Chip support
CONFIG_I2C_HELPER_AUTO=yCONFIG_I2C_SMBUS=mCONFIG_I2C_ALGOBIT=mCONFIG_I2C_ALGOPCA=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#CONFIG_I2C_CCGX_UCSI=mCONFIG_I2C_ALI1535=mCONFIG_I2C_ALI1563=mCONFIG_I2C_ALI15X3=mCONFIG_I2C_AMD756=mCONFIG_I2C_AMD756_S4882=mCONFIG_I2C_AMD8111=mCONFIG_I2C_AMD_MP2=mCONFIG_I2C_I801=mCONFIG_I2C_ISCH=mCONFIG_I2C_ISMT=mCONFIG_I2C_PIIX4=yCONFIG_I2C_CHT_WC=mCONFIG_I2C_NFORCE2=mCONFIG_I2C_NFORCE2_S4985=mCONFIG_I2C_NVIDIA_GPU=mCONFIG_I2C_SIS5595=mCONFIG_I2C_SIS630=mCONFIG_I2C_SIS96X=mCONFIG_I2C_VIA=mCONFIG_I2C_VIAPRO=m
#
# ACPI drivers
#CONFIG_I2C_SCMI=m
#
# I2C system bus drivers (mostly embedded / system-on-chip) #CONFIG_I2C_CBUS_GPIO=m CONFIG_I2C_DESIGNWARE_CORE=y
# CONFIG_I2C_DESIGNWARE_SLAVE is not set CONFIG_I2C_DESIGNWARE_PLATFORM=y CONFIG_I2C_DESIGNWARE_BAYTRAIL=y CONFIG_I2C_DESIGNWARE_PCI=m
# CONFIG_I2C_EMEV2 is not setCONFIG_I2C_GPIO=m
# CONFIG_I2C_GPIO_FAULT_INJECTOR is not set CONFIG_I2C_KEMPLD=mCONFIG_I2C_OCORES=mCONFIG_I2C_PCA_PLATFORM=mCONFIG_I2C_SIMTEC=mCONFIG_I2C_XILINX=m
#
# External I2C/SMBus adapter drivers
#CONFIG_I2C_DIOLAN_U2C=mCONFIG_I2C_DLN2=mCONFIG_I2C_CP2615=mCONFIG_I2C_PARPORT=m
# CONFIG_I2C_PCI1XXXX is not setCONFIG_I2C_ROBOTFUZZ_OSIF=mCONFIG_I2C_TAOS_EVM=mCONFIG_I2C_TINY_USB=mCONFIG_I2C_VIPERBOARD=m
#
# Other I2C/SMBus bus drivers
#CONFIG_I2C_MLXCPLD=mCONFIG_I2C_CROS_EC_TUNNEL=mCONFIG_I2C_VIRTIO=m
# end of I2C Hardware Bus support
CONFIG_I2C_STUB=m
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# end of I2C support
CONFIG_I3C=mCONFIG_CDNS_I3C_MASTER=mCONFIG_DW_I3C_MASTER=mCONFIG_SVC_I3C_MASTER=mCONFIG_MIPI_I3C_HCI=mCONFIG_SPI=y
# CONFIG_SPI_DEBUG is not setCONFIG_SPI_MASTER=yCONFIG_SPI_MEM=y
#
# SPI Master Controller Drivers
#CONFIG_SPI_ALTERA=mCONFIG_SPI_ALTERA_CORE=mCONFIG_SPI_ALTERA_DFL=mCONFIG_SPI_AXI_SPI_ENGINE=mCONFIG_SPI_BITBANG=mCONFIG_SPI_BUTTERFLY=mCONFIG_SPI_CADENCE=mCONFIG_SPI_DESIGNWARE=mCONFIG_SPI_DW_DMA=yCONFIG_SPI_DW_PCI=mCONFIG_SPI_DW_MMIO=mCONFIG_SPI_DLN2=mCONFIG_SPI_GPIO=m
# CONFIG_SPI_INTEL_PCI is not set
# CONFIG_SPI_INTEL_PLATFORM is not setCONFIG_SPI_LM70_LLP=m
# CONFIG_SPI_MICROCHIP_CORE is not set
# CONFIG_SPI_MICROCHIP_CORE_QSPI is not setCONFIG_SPI_LANTIQ_SSC=mCONFIG_SPI_OC_TINY=m
# CONFIG_SPI_PCI1XXXX is not setCONFIG_SPI_PXA2XX=mCONFIG_SPI_PXA2XX_PCI=mCONFIG_SPI_SC18IS602=mCONFIG_SPI_SIFIVE=mCONFIG_SPI_MXIC=mCONFIG_SPI_XCOMM=m
# CONFIG_SPI_XILINX is not setCONFIG_SPI_ZYNQMP_GQSPI=mCONFIG_SPI_AMD=m
#
# SPI Multiplexer support
#CONFIG_SPI_MUX=m
#
# SPI Protocol Masters
#CONFIG_SPI_SPIDEV=mCONFIG_SPI_LOOPBACK_TEST=mCONFIG_SPI_TLE62X0=mCONFIG_SPI_SLAVE=yCONFIG_SPI_SLAVE_TIME=mCONFIG_SPI_SLAVE_SYSTEM_CONTROL=mCONFIG_SPI_DYNAMIC=yCONFIG_SPMI=mCONFIG_SPMI_HISI3670=mCONFIG_HSI=mCONFIG_HSI_BOARDINFO=y
#
# HSI controllers
#
#
# HSI clients
#CONFIG_HSI_CHAR=mCONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not setCONFIG_PPS_CLIENT_LDISC=mCONFIG_PPS_CLIENT_PARPORT=mCONFIG_PPS_CLIENT_GPIO=m
#
# PPS generators support
#
#
# PTP clock support
#CONFIG_PTP_1588_CLOCK=yCONFIG_PTP_1588_CLOCK_OPTIONAL=yCONFIG_DP83640_PHY=mCONFIG_PTP_1588_CLOCK_INES=mCONFIG_PTP_1588_CLOCK_KVM=mCONFIG_PTP_1588_CLOCK_IDT82P33=mCONFIG_PTP_1588_CLOCK_IDTCM=m
# CONFIG_PTP_1588_CLOCK_FC3W is not set
# CONFIG_PTP_1588_CLOCK_MOCK is not setCONFIG_PTP_1588_CLOCK_VMW=mCONFIG_PTP_1588_CLOCK_OCP=m
# CONFIG_PTP_DFL_TOD is not set
# end of PTP clock support
CONFIG_PINCTRL=yCONFIG_PINMUX=yCONFIG_PINCONF=yCONFIG_GENERIC_PINCONF=y
# CONFIG_DEBUG_PINCTRL is not setCONFIG_PINCTRL_AMD=y
# CONFIG_PINCTRL_CY8C95X0 is not setCONFIG_PINCTRL_DA9062=mCONFIG_PINCTRL_MCP23S08_I2C=mCONFIG_PINCTRL_MCP23S08_SPI=mCONFIG_PINCTRL_MCP23S08=mCONFIG_PINCTRL_SX150X=y
# CONFIG_PINCTRL_CS42L43 is not setCONFIG_PINCTRL_MADERA=mCONFIG_PINCTRL_CS47L15=yCONFIG_PINCTRL_CS47L35=yCONFIG_PINCTRL_CS47L85=yCONFIG_PINCTRL_CS47L90=yCONFIG_PINCTRL_CS47L92=y
#
# Intel pinctrl drivers
#CONFIG_PINCTRL_BAYTRAIL=yCONFIG_PINCTRL_CHERRYVIEW=yCONFIG_PINCTRL_LYNXPOINT=mCONFIG_PINCTRL_INTEL=y
# CONFIG_PINCTRL_INTEL_PLATFORM is not setCONFIG_PINCTRL_ALDERLAKE=mCONFIG_PINCTRL_BROXTON=mCONFIG_PINCTRL_CANNONLAKE=mCONFIG_PINCTRL_CEDARFORK=mCONFIG_PINCTRL_DENVERTON=mCONFIG_PINCTRL_ELKHARTLAKE=mCONFIG_PINCTRL_EMMITSBURG=mCONFIG_PINCTRL_GEMINILAKE=mCONFIG_PINCTRL_ICELAKE=mCONFIG_PINCTRL_JASPERLAKE=mCONFIG_PINCTRL_LAKEFIELD=mCONFIG_PINCTRL_LEWISBURG=m
# CONFIG_PINCTRL_METEORLAKE is not set
# CONFIG_PINCTRL_METEORPOINT is not setCONFIG_PINCTRL_SUNRISEPOINT=mCONFIG_PINCTRL_TIGERLAKE=m
# end of Intel pinctrl drivers
#
# Renesas pinctrl drivers
#
# end of Renesas pinctrl drivers
CONFIG_GPIOLIB=yCONFIG_GPIOLIB_FASTPATH_LIMIT=512CONFIG_GPIO_ACPI=yCONFIG_GPIOLIB_IRQCHIP=y
# CONFIG_DEBUG_GPIO is not setCONFIG_GPIO_SYSFS=yCONFIG_GPIO_CDEV=y
# CONFIG_GPIO_CDEV_V1 is not setCONFIG_GPIO_GENERIC=mCONFIG_GPIO_REGMAP=mCONFIG_GPIO_MAX730X=mCONFIG_GPIO_IDIO_16=m
#
# Memory mapped GPIO drivers
#CONFIG_GPIO_AMDPT=mCONFIG_GPIO_DWAPB=mCONFIG_GPIO_EXAR=mCONFIG_GPIO_GENERIC_PLATFORM=mCONFIG_GPIO_ICH=mCONFIG_GPIO_MB86S7X=mCONFIG_GPIO_MENZ127=mCONFIG_GPIO_SIOX=mCONFIG_GPIO_AMD_FCH=m
# end of Memory mapped GPIO drivers
#
# Port-mapped I/O GPIO drivers
#CONFIG_GPIO_VX855=mCONFIG_GPIO_I8255=mCONFIG_GPIO_104_DIO_48E=mCONFIG_GPIO_104_IDIO_16=mCONFIG_GPIO_104_IDI_48=mCONFIG_GPIO_F7188X=mCONFIG_GPIO_GPIO_MM=mCONFIG_GPIO_IT87=mCONFIG_GPIO_SCH=mCONFIG_GPIO_SCH311X=mCONFIG_GPIO_WINBOND=mCONFIG_GPIO_WS16C48=m
# end of Port-mapped I/O GPIO drivers
#
# I2C GPIO expanders
#
# CONFIG_GPIO_FXL6408 is not set
# CONFIG_GPIO_DS4520 is not setCONFIG_GPIO_MAX7300=mCONFIG_GPIO_MAX732X=mCONFIG_GPIO_PCA953X=mCONFIG_GPIO_PCA953X_IRQ=yCONFIG_GPIO_PCA9570=mCONFIG_GPIO_PCF857X=mCONFIG_GPIO_TPIC2810=m
# end of I2C GPIO expanders
#
# MFD GPIO expanders
#CONFIG_GPIO_ADP5520=mCONFIG_GPIO_ARIZONA=mCONFIG_GPIO_BD9571MWV=m
# CONFIG_GPIO_CROS_EC is not setCONFIG_GPIO_CRYSTAL_COVE=yCONFIG_GPIO_DA9052=mCONFIG_GPIO_DA9055=mCONFIG_GPIO_DLN2=m
# CONFIG_GPIO_ELKHARTLAKE is not setCONFIG_GPIO_JANZ_TTL=mCONFIG_GPIO_KEMPLD=mCONFIG_GPIO_LP3943=mCONFIG_GPIO_LP873X=mCONFIG_GPIO_MADERA=mCONFIG_GPIO_PALMAS=yCONFIG_GPIO_RC5T583=yCONFIG_GPIO_TPS65086=mCONFIG_GPIO_TPS6586X=yCONFIG_GPIO_TPS65910=yCONFIG_GPIO_TPS65912=mCONFIG_GPIO_TPS68470=mCONFIG_GPIO_TQMX86=mCONFIG_GPIO_TWL4030=mCONFIG_GPIO_TWL6040=mCONFIG_GPIO_WHISKEY_COVE=mCONFIG_GPIO_WM831X=mCONFIG_GPIO_WM8350=mCONFIG_GPIO_WM8994=m
# end of MFD GPIO expanders
#
# PCI GPIO expanders
#CONFIG_GPIO_AMD8111=mCONFIG_GPIO_ML_IOH=mCONFIG_GPIO_PCI_IDIO_16=mCONFIG_GPIO_PCIE_IDIO_24=mCONFIG_GPIO_RDC321X=m
# end of PCI GPIO expanders
#
# SPI GPIO expanders
#CONFIG_GPIO_MAX3191X=mCONFIG_GPIO_MAX7301=mCONFIG_GPIO_MC33880=mCONFIG_GPIO_PISOSR=mCONFIG_GPIO_XRA1403=m
# end of SPI GPIO expanders
#
# USB GPIO expanders
#CONFIG_GPIO_VIPERBOARD=m
# end of USB GPIO expanders
#
# Virtual GPIO drivers
#CONFIG_GPIO_AGGREGATOR=m
# CONFIG_GPIO_LATCH is not set
# CONFIG_GPIO_MOCKUP is not setCONFIG_GPIO_VIRTIO=m
# CONFIG_GPIO_SIM is not set
# end of Virtual GPIO drivers
CONFIG_W1=mCONFIG_W1_CON=y
#
# 1-wire Bus Masters
#
# CONFIG_W1_MASTER_AMD_AXI is not setCONFIG_W1_MASTER_MATROX=mCONFIG_W1_MASTER_DS2490=mCONFIG_W1_MASTER_DS2482=mCONFIG_W1_MASTER_GPIO=mCONFIG_W1_MASTER_SGI=m
# CONFIG_W1_MASTER_UART is not set
# end of 1-wire Bus Masters
#
# 1-wire Slaves
#CONFIG_W1_SLAVE_THERM=mCONFIG_W1_SLAVE_SMEM=mCONFIG_W1_SLAVE_DS2405=mCONFIG_W1_SLAVE_DS2408=mCONFIG_W1_SLAVE_DS2408_READBACK=yCONFIG_W1_SLAVE_DS2413=mCONFIG_W1_SLAVE_DS2406=mCONFIG_W1_SLAVE_DS2423=mCONFIG_W1_SLAVE_DS2805=mCONFIG_W1_SLAVE_DS2430=mCONFIG_W1_SLAVE_DS2431=mCONFIG_W1_SLAVE_DS2433=m
# CONFIG_W1_SLAVE_DS2433_CRC is not setCONFIG_W1_SLAVE_DS2438=mCONFIG_W1_SLAVE_DS250X=mCONFIG_W1_SLAVE_DS2780=mCONFIG_W1_SLAVE_DS2781=mCONFIG_W1_SLAVE_DS28E04=mCONFIG_W1_SLAVE_DS28E17=m
# end of 1-wire Slaves
CONFIG_POWER_RESET=y CONFIG_POWER_RESET_ATC260X=m CONFIG_POWER_RESET_MT6323=y CONFIG_POWER_RESET_RESTART=y CONFIG_POWER_RESET_TPS65086=y CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set CONFIG_POWER_SUPPLY_HWMON=y CONFIG_GENERIC_ADC_BATTERY=m
# CONFIG_IP5XXX_POWER is not set CONFIG_MAX8925_POWER=m CONFIG_WM831X_BACKUP=m CONFIG_WM831X_POWER=m CONFIG_WM8350_POWER=mCONFIG_TEST_POWER=m CONFIG_BATTERY_88PM860X=m CONFIG_CHARGER_ADP5061=m CONFIG_BATTERY_CW2015=m CONFIG_BATTERY_DS2760=m CONFIG_BATTERY_DS2780=m CONFIG_BATTERY_DS2781=m CONFIG_BATTERY_DS2782=m
# CONFIG_BATTERY_SAMSUNG_SDI is not set CONFIG_BATTERY_SBS=mCONFIG_CHARGER_SBS=mCONFIG_MANAGER_SBS=m CONFIG_BATTERY_BQ27XXX=m CONFIG_BATTERY_BQ27XXX_I2C=m CONFIG_BATTERY_BQ27XXX_HDQ=m
# CONFIG_BATTERY_BQ27XXX_DT_UPDATES_NVM is not set CONFIG_BATTERY_DA9030=m CONFIG_BATTERY_DA9052=m CONFIG_CHARGER_DA9150=m CONFIG_BATTERY_DA9150=m CONFIG_CHARGER_AXP20X=m CONFIG_BATTERY_AXP20X=m CONFIG_AXP20X_POWER=m CONFIG_AXP288_CHARGER=m CONFIG_AXP288_FUEL_GAUGE=m CONFIG_BATTERY_MAX17040=m CONFIG_BATTERY_MAX17042=m CONFIG_BATTERY_MAX1721X=m CONFIG_BATTERY_TWL4030_MADC=m CONFIG_CHARGER_88PM860X=m CONFIG_CHARGER_PCF50633=m CONFIG_BATTERY_RX51=m CONFIG_CHARGER_ISP1704=m CONFIG_CHARGER_MAX8903=m CONFIG_CHARGER_TWL4030=m CONFIG_CHARGER_LP8727=m CONFIG_CHARGER_LP8788=m CONFIG_CHARGER_GPIO=m CONFIG_CHARGER_MANAGER=y CONFIG_CHARGER_LT3651=m CONFIG_CHARGER_LTC4162L=m CONFIG_CHARGER_MAX14577=m CONFIG_CHARGER_MAX77693=m
# CONFIG_CHARGER_MAX77976 is not set CONFIG_CHARGER_MAX8997=m CONFIG_CHARGER_MAX8998=m CONFIG_CHARGER_MP2629=m CONFIG_CHARGER_MT6360=m CONFIG_CHARGER_BQ2415X=m CONFIG_CHARGER_BQ24190=m CONFIG_CHARGER_BQ24257=m CONFIG_CHARGER_BQ24735=m CONFIG_CHARGER_BQ2515X=m CONFIG_CHARGER_BQ25890=m CONFIG_CHARGER_BQ25980=m CONFIG_CHARGER_BQ256XX=m CONFIG_CHARGER_SMB347=m CONFIG_CHARGER_TPS65090=m CONFIG_BATTERY_GAUGE_LTC2941=m CONFIG_BATTERY_GOLDFISH=m CONFIG_BATTERY_RT5033=m
# CONFIG_CHARGER_RT5033 is not set CONFIG_CHARGER_RT9455=m
# CONFIG_CHARGER_RT9467 is not set
# CONFIG_CHARGER_RT9471 is not set CONFIG_CHARGER_CROS_USBPD=m CONFIG_CHARGER_CROS_PCHG=m CONFIG_CHARGER_BD99954=m CONFIG_CHARGER_WILCO=m CONFIG_BATTERY_SURFACE=m CONFIG_CHARGER_SURFACE=m
# CONFIG_BATTERY_UG3105 is not set
# CONFIG_FUEL_GAUGE_MM8013 is not set CONFIG_HWMON=yCONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#CONFIG_SENSORS_ABITUGURU=mCONFIG_SENSORS_ABITUGURU3=mCONFIG_SENSORS_AD7314=mCONFIG_SENSORS_AD7414=mCONFIG_SENSORS_AD7418=mCONFIG_SENSORS_ADM1025=mCONFIG_SENSORS_ADM1026=mCONFIG_SENSORS_ADM1029=mCONFIG_SENSORS_ADM1031=mCONFIG_SENSORS_ADM1177=mCONFIG_SENSORS_ADM9240=mCONFIG_SENSORS_ADT7X10=mCONFIG_SENSORS_ADT7310=mCONFIG_SENSORS_ADT7410=mCONFIG_SENSORS_ADT7411=mCONFIG_SENSORS_ADT7462=mCONFIG_SENSORS_ADT7470=mCONFIG_SENSORS_ADT7475=mCONFIG_SENSORS_AHT10=mCONFIG_SENSORS_AQUACOMPUTER_D5NEXT=mCONFIG_SENSORS_AS370=mCONFIG_SENSORS_ASC7621=m
# CONFIG_SENSORS_ASUS_ROG_RYUJIN is not setCONFIG_SENSORS_AXI_FAN_CONTROL=mCONFIG_SENSORS_K8TEMP=mCONFIG_SENSORS_K10TEMP=mCONFIG_SENSORS_FAM15H_POWER=mCONFIG_SENSORS_APPLESMC=mCONFIG_SENSORS_ASB100=mCONFIG_SENSORS_ATXP1=m
# CONFIG_SENSORS_CHIPCAP2 is not setCONFIG_SENSORS_CORSAIR_CPRO=mCONFIG_SENSORS_CORSAIR_PSU=mCONFIG_SENSORS_DRIVETEMP=mCONFIG_SENSORS_DS620=mCONFIG_SENSORS_DS1621=mCONFIG_SENSORS_DELL_SMM=m
# CONFIG_I8K is not setCONFIG_SENSORS_DA9052_ADC=mCONFIG_SENSORS_DA9055=mCONFIG_SENSORS_I5K_AMB=mCONFIG_SENSORS_F71805F=mCONFIG_SENSORS_F71882FG=mCONFIG_SENSORS_F75375S=mCONFIG_SENSORS_MC13783_ADC=mCONFIG_SENSORS_FSCHMD=mCONFIG_SENSORS_FTSTEUTATES=m
# CONFIG_SENSORS_GIGABYTE_WATERFORCE is not setCONFIG_SENSORS_GL518SM=mCONFIG_SENSORS_GL520SM=mCONFIG_SENSORS_G760A=mCONFIG_SENSORS_G762=mCONFIG_SENSORS_HIH6130=m
# CONFIG_SENSORS_HS3001 is not setCONFIG_SENSORS_IBMAEM=mCONFIG_SENSORS_IBMPEX=mCONFIG_SENSORS_IIO_HWMON=mCONFIG_SENSORS_I5500=mCONFIG_SENSORS_CORETEMP=mCONFIG_SENSORS_IT87=mCONFIG_SENSORS_JC42=m
# CONFIG_SENSORS_POWERZ is not setCONFIG_SENSORS_POWR1220=mCONFIG_SENSORS_LINEAGE=mCONFIG_SENSORS_LTC2945=mCONFIG_SENSORS_LTC2947=mCONFIG_SENSORS_LTC2947_I2C=mCONFIG_SENSORS_LTC2947_SPI=mCONFIG_SENSORS_LTC2990=m
# CONFIG_SENSORS_LTC2991 is not setCONFIG_SENSORS_LTC2992=mCONFIG_SENSORS_LTC4151=mCONFIG_SENSORS_LTC4215=mCONFIG_SENSORS_LTC4222=mCONFIG_SENSORS_LTC4245=mCONFIG_SENSORS_LTC4260=mCONFIG_SENSORS_LTC4261=m
# CONFIG_SENSORS_LTC4282 is not setCONFIG_SENSORS_MAX1111=mCONFIG_SENSORS_MAX127=mCONFIG_SENSORS_MAX16065=mCONFIG_SENSORS_MAX1619=mCONFIG_SENSORS_MAX1668=mCONFIG_SENSORS_MAX197=mCONFIG_SENSORS_MAX31722=mCONFIG_SENSORS_MAX31730=m
# CONFIG_SENSORS_MAX31760 is not set
# CONFIG_MAX31827 is not set
# CONFIG_SENSORS_MAX6620 is not setCONFIG_SENSORS_MAX6621=mCONFIG_SENSORS_MAX6639=mCONFIG_SENSORS_MAX6650=mCONFIG_SENSORS_MAX6697=mCONFIG_SENSORS_MAX31790=m
# CONFIG_SENSORS_MC34VR500 is not setCONFIG_SENSORS_MCP3021=mCONFIG_SENSORS_MLXREG_FAN=mCONFIG_SENSORS_TC654=mCONFIG_SENSORS_TPS23861=mCONFIG_SENSORS_MENF21BMC_HWMON=mCONFIG_SENSORS_MR75203=mCONFIG_SENSORS_ADCXX=mCONFIG_SENSORS_LM63=mCONFIG_SENSORS_LM70=mCONFIG_SENSORS_LM73=mCONFIG_SENSORS_LM75=mCONFIG_SENSORS_LM77=mCONFIG_SENSORS_LM78=mCONFIG_SENSORS_LM80=mCONFIG_SENSORS_LM83=mCONFIG_SENSORS_LM85=mCONFIG_SENSORS_LM87=mCONFIG_SENSORS_LM90=mCONFIG_SENSORS_LM92=mCONFIG_SENSORS_LM93=mCONFIG_SENSORS_LM95234=mCONFIG_SENSORS_LM95241=mCONFIG_SENSORS_LM95245=mCONFIG_SENSORS_PC87360=mCONFIG_SENSORS_PC87427=mCONFIG_SENSORS_NTC_THERMISTOR=mCONFIG_SENSORS_NCT6683=mCONFIG_SENSORS_NCT6775_CORE=mCONFIG_SENSORS_NCT6775=m
# CONFIG_SENSORS_NCT6775_I2C is not setCONFIG_SENSORS_NCT7802=mCONFIG_SENSORS_NCT7904=mCONFIG_SENSORS_NPCM7XX=mCONFIG_SENSORS_NZXT_KRAKEN2=m
# CONFIG_SENSORS_NZXT_KRAKEN3 is not set
# CONFIG_SENSORS_NZXT_SMART2 is not set
# CONFIG_SENSORS_OCC_P8_I2C is not set
# CONFIG_SENSORS_OXP is not setCONFIG_SENSORS_PCF8591=mCONFIG_PMBUS=mCONFIG_SENSORS_PMBUS=m
# CONFIG_SENSORS_ACBEL_FSG032 is not setCONFIG_SENSORS_ADM1266=mCONFIG_SENSORS_ADM1275=mCONFIG_SENSORS_BEL_PFE=mCONFIG_SENSORS_BPA_RS600=m
# CONFIG_SENSORS_DELTA_AHE50DC_FAN is not setCONFIG_SENSORS_FSP_3Y=mCONFIG_SENSORS_IBM_CFFPS=mCONFIG_SENSORS_DPS920AB=mCONFIG_SENSORS_INSPUR_IPSPS=mCONFIG_SENSORS_IR35221=mCONFIG_SENSORS_IR36021=mCONFIG_SENSORS_IR38064=m
# CONFIG_SENSORS_IR38064_REGULATOR is not setCONFIG_SENSORS_IRPS5401=mCONFIG_SENSORS_ISL68137=mCONFIG_SENSORS_LM25066=m
# CONFIG_SENSORS_LM25066_REGULATOR is not set
# CONFIG_SENSORS_LT7182S is not setCONFIG_SENSORS_LTC2978=mCONFIG_SENSORS_LTC2978_REGULATOR=yCONFIG_SENSORS_LTC3815=m
# CONFIG_SENSORS_LTC4286 is not setCONFIG_SENSORS_MAX15301=mCONFIG_SENSORS_MAX16064=mCONFIG_SENSORS_MAX16601=mCONFIG_SENSORS_MAX20730=mCONFIG_SENSORS_MAX20751=mCONFIG_SENSORS_MAX31785=mCONFIG_SENSORS_MAX34440=mCONFIG_SENSORS_MAX8688=m
# CONFIG_SENSORS_MP2856 is not setCONFIG_SENSORS_MP2888=mCONFIG_SENSORS_MP2975=m
# CONFIG_SENSORS_MP2975_REGULATOR is not set
# CONFIG_SENSORS_MP5023 is not set
# CONFIG_SENSORS_MP5990 is not set
# CONFIG_SENSORS_MPQ7932 is not set
# CONFIG_SENSORS_MPQ8785 is not setCONFIG_SENSORS_PIM4328=m
# CONFIG_SENSORS_PLI1209BC is not setCONFIG_SENSORS_PM6764TR=mCONFIG_SENSORS_PXE1610=mCONFIG_SENSORS_Q54SJ108A2=mCONFIG_SENSORS_STPDDC60=m
# CONFIG_SENSORS_TDA38640 is not setCONFIG_SENSORS_TPS40422=mCONFIG_SENSORS_TPS53679=m
# CONFIG_SENSORS_TPS546D24 is not setCONFIG_SENSORS_UCD9000=mCONFIG_SENSORS_UCD9200=m
# CONFIG_SENSORS_XDPE152 is not setCONFIG_SENSORS_XDPE122=m
# CONFIG_SENSORS_XDPE122_REGULATOR is not setCONFIG_SENSORS_ZL6100=m
# CONFIG_SENSORS_PT5161L is not setCONFIG_SENSORS_SBTSI=mCONFIG_SENSORS_SBRMI=mCONFIG_SENSORS_SHT15=mCONFIG_SENSORS_SHT21=m
CONFIG_SENSORS_SHT3x=m
CONFIG_SENSORS_SHT4x=mCONFIG_SENSORS_SHTC1=mCONFIG_SENSORS_SIS5595=mCONFIG_SENSORS_DME1737=mCONFIG_SENSORS_EMC1403=mCONFIG_SENSORS_EMC2103=m
# CONFIG_SENSORS_EMC2305 is not setCONFIG_SENSORS_EMC6W201=mCONFIG_SENSORS_SMSC47M1=mCONFIG_SENSORS_SMSC47M192=mCONFIG_SENSORS_SMSC47B397=mCONFIG_SENSORS_SCH56XX_COMMON=mCONFIG_SENSORS_SCH5627=mCONFIG_SENSORS_SCH5636=mCONFIG_SENSORS_STTS751=m
# CONFIG_SENSORS_SURFACE_FAN is not setCONFIG_SENSORS_ADC128D818=mCONFIG_SENSORS_ADS7828=mCONFIG_SENSORS_ADS7871=mCONFIG_SENSORS_AMC6821=mCONFIG_SENSORS_INA209=mCONFIG_SENSORS_INA2XX=m
# CONFIG_SENSORS_INA238 is not setCONFIG_SENSORS_INA3221=mCONFIG_SENSORS_TC74=mCONFIG_SENSORS_THMC50=mCONFIG_SENSORS_TMP102=mCONFIG_SENSORS_TMP103=mCONFIG_SENSORS_TMP108=mCONFIG_SENSORS_TMP401=mCONFIG_SENSORS_TMP421=m
# CONFIG_SENSORS_TMP464 is not setCONFIG_SENSORS_TMP513=mCONFIG_SENSORS_VIA_CPUTEMP=mCONFIG_SENSORS_VIA686A=mCONFIG_SENSORS_VT1211=mCONFIG_SENSORS_VT8231=mCONFIG_SENSORS_W83773G=mCONFIG_SENSORS_W83781D=mCONFIG_SENSORS_W83791D=mCONFIG_SENSORS_W83792D=mCONFIG_SENSORS_W83793=mCONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not setCONFIG_SENSORS_W83L785TS=mCONFIG_SENSORS_W83L786NG=mCONFIG_SENSORS_W83627HF=mCONFIG_SENSORS_W83627EHF=mCONFIG_SENSORS_WM831X=mCONFIG_SENSORS_WM8350=mCONFIG_SENSORS_XGENE=m
#
# ACPI drivers
#CONFIG_SENSORS_ACPI_POWER=m CONFIG_SENSORS_ATK0110=m
# CONFIG_SENSORS_ASUS_WMI is not set
# CONFIG_SENSORS_ASUS_EC is not set
# CONFIG_SENSORS_HP_WMI is not set CONFIG_THERMAL=yCONFIG_THERMAL_NETLINK=y CONFIG_THERMAL_STATISTICS=y
# CONFIG_THERMAL_DEBUGFS is not set CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0 CONFIG_THERMAL_HWMON=y CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set # CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set # CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set # CONFIG_THERMAL_DEFAULT_GOV_BANG_BANG is not set CONFIG_THERMAL_GOV_FAIR_SHARE=y CONFIG_THERMAL_GOV_STEP_WISE=y CONFIG_THERMAL_GOV_BANG_BANG=y CONFIG_THERMAL_GOV_USER_SPACE=y CONFIG_THERMAL_GOV_POWER_ALLOCATOR=y CONFIG_DEVFREQ_THERMAL=y CONFIG_THERMAL_EMULATION=y
#
# Intel thermal drivers
#CONFIG_INTEL_POWERCLAMP=mCONFIG_X86_THERMAL_VECTOR=yCONFIG_INTEL_TCC=yCONFIG_X86_PKG_TEMP_THERMAL=mCONFIG_INTEL_SOC_DTS_IOSF_CORE=mCONFIG_INTEL_SOC_DTS_THERMAL=m
#
# ACPI INT340X thermal drivers
#CONFIG_INT340X_THERMAL=mCONFIG_ACPI_THERMAL_REL=mCONFIG_INT3406_THERMAL=mCONFIG_PROC_THERMAL_MMIO_RAPL=m
# end of ACPI INT340X thermal drivers
CONFIG_INTEL_BXT_PMIC_THERMAL=mCONFIG_INTEL_PCH_THERMAL=mCONFIG_INTEL_TCC_COOLING=m
# CONFIG_INTEL_HFI_THERMAL is not set
# end of Intel thermal drivers
CONFIG_GENERIC_ADC_THERMAL=mCONFIG_WATCHDOG=yCONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not setCONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=yCONFIG_WATCHDOG_OPEN_TIMEOUT=0CONFIG_WATCHDOG_SYSFS=y
# CONFIG_WATCHDOG_HRTIMER_PRETIMEOUT is not set
#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set
#
# Watchdog Device Drivers
#CONFIG_SOFT_WATCHDOG=m
# CONFIG_CROS_EC_WATCHDOG is not setCONFIG_DA9052_WATCHDOG=mCONFIG_DA9055_WATCHDOG=mCONFIG_DA9063_WATCHDOG=mCONFIG_DA9062_WATCHDOG=mCONFIG_MENF21BMC_WATCHDOG=mCONFIG_MENZ069_WATCHDOG=mCONFIG_WDAT_WDT=mCONFIG_WM831X_WATCHDOG=mCONFIG_WM8350_WATCHDOG=mCONFIG_XILINX_WATCHDOG=mCONFIG_ZIIRAVE_WATCHDOG=mCONFIG_RAVE_SP_WATCHDOG=mCONFIG_MLX_WDT=mCONFIG_CADENCE_WATCHDOG=mCONFIG_DW_WATCHDOG=mCONFIG_TWL4030_WATCHDOG=mCONFIG_MAX63XX_WATCHDOG=mCONFIG_RETU_WATCHDOG=mCONFIG_ACQUIRE_WDT=mCONFIG_ADVANTECH_WDT=m
# CONFIG_ADVANTECH_EC_WDT is not setCONFIG_ALIM1535_WDT=mCONFIG_ALIM7101_WDT=mCONFIG_EBC_C384_WDT=m
# CONFIG_EXAR_WDT is not setCONFIG_F71808E_WDT=mCONFIG_SP5100_TCO=yCONFIG_SBC_FITPC2_WATCHDOG=mCONFIG_EUROTECH_WDT=mCONFIG_IB700_WDT=mCONFIG_IBMASR=mCONFIG_WAFER_WDT=mCONFIG_I6300ESB_WDT=mCONFIG_IE6XX_WDT=mCONFIG_ITCO_WDT=mCONFIG_ITCO_VENDOR_SUPPORT=yCONFIG_IT8712F_WDT=mCONFIG_IT87_WDT=mCONFIG_HP_WATCHDOG=mCONFIG_HPWDT_NMI_DECODING=yCONFIG_KEMPLD_WDT=mCONFIG_SC1200_WDT=mCONFIG_PC87413_WDT=mCONFIG_NV_TCO=mCONFIG_60XX_WDT=mCONFIG_CPU5_WDT=mCONFIG_SMSC_SCH311X_WDT=mCONFIG_SMSC37B787_WDT=mCONFIG_TQMX86_WDT=mCONFIG_VIA_WDT=mCONFIG_W83627HF_WDT=mCONFIG_W83877F_WDT=mCONFIG_W83977F_WDT=mCONFIG_MACHZ_WDT=mCONFIG_SBC_EPX_C3_WATCHDOG=mCONFIG_INTEL_MEI_WDT=mCONFIG_NI903X_WDT=mCONFIG_NIC7018_WDT=mCONFIG_MEN_A21_WDT=mCONFIG_XEN_WDT=m
#
# PCI-based Watchdog Cards
#CONFIG_PCIPCWATCHDOG=mCONFIG_WDTPCI=m
#
# USB-based Watchdog Cards
#CONFIG_USBPCWATCHDOG=mCONFIG_SSB_POSSIBLE=yCONFIG_SSB=mCONFIG_SSB_SPROM=yCONFIG_SSB_BLOCKIO=yCONFIG_SSB_PCIHOST_POSSIBLE=yCONFIG_SSB_PCIHOST=yCONFIG_SSB_B43_PCI_BRIDGE=yCONFIG_SSB_PCMCIAHOST_POSSIBLE=y
# CONFIG_SSB_PCMCIAHOST is not setCONFIG_SSB_SDIOHOST_POSSIBLE=yCONFIG_SSB_SDIOHOST=yCONFIG_SSB_DRIVER_PCICORE_POSSIBLE=yCONFIG_SSB_DRIVER_PCICORE=yCONFIG_SSB_DRIVER_GPIO=yCONFIG_BCMA_POSSIBLE=yCONFIG_BCMA=mCONFIG_BCMA_BLOCKIO=yCONFIG_BCMA_HOST_PCI_POSSIBLE=yCONFIG_BCMA_HOST_PCI=yCONFIG_BCMA_HOST_SOC=yCONFIG_BCMA_DRIVER_PCI=yCONFIG_BCMA_SFLASH=yCONFIG_BCMA_DRIVER_GMAC_CMN=yCONFIG_BCMA_DRIVER_GPIO=y
# CONFIG_BCMA_DEBUG is not set
#
# Multifunction device drivers
#CONFIG_MFD_CORE=yCONFIG_MFD_AS3711=y
# CONFIG_MFD_SMPRO is not setCONFIG_PMIC_ADP5520=yCONFIG_MFD_AAT2870_CORE=yCONFIG_MFD_BCM590XX=mCONFIG_MFD_BD9571MWV=mCONFIG_MFD_AXP20X=mCONFIG_MFD_AXP20X_I2C=mCONFIG_MFD_CROS_EC_DEV=mCONFIG_MFD_CS42L43=m
# CONFIG_MFD_CS42L43_I2C is not setCONFIG_MFD_CS42L43_SDW=mCONFIG_MFD_MADERA=mCONFIG_MFD_MADERA_I2C=mCONFIG_MFD_MADERA_SPI=mCONFIG_MFD_CS47L15=yCONFIG_MFD_CS47L35=yCONFIG_MFD_CS47L85=yCONFIG_MFD_CS47L90=yCONFIG_MFD_CS47L92=yCONFIG_PMIC_DA903X=yCONFIG_PMIC_DA9052=yCONFIG_MFD_DA9052_SPI=yCONFIG_MFD_DA9052_I2C=yCONFIG_MFD_DA9055=yCONFIG_MFD_DA9062=mCONFIG_MFD_DA9063=yCONFIG_MFD_DA9150=mCONFIG_MFD_DLN2=mCONFIG_MFD_MC13XXX=mCONFIG_MFD_MC13XXX_SPI=mCONFIG_MFD_MC13XXX_I2C=mCONFIG_MFD_MP2629=mCONFIG_MFD_INTEL_QUARK_I2C_GPIO=mCONFIG_LPC_ICH=mCONFIG_LPC_SCH=mCONFIG_INTEL_SOC_PMIC=yCONFIG_INTEL_SOC_PMIC_BXTWC=mCONFIG_INTEL_SOC_PMIC_CHTWC=yCONFIG_INTEL_SOC_PMIC_CHTDC_TI=mCONFIG_INTEL_SOC_PMIC_MRFLD=mCONFIG_MFD_INTEL_LPSS=mCONFIG_MFD_INTEL_LPSS_ACPI=mCONFIG_MFD_INTEL_LPSS_PCI=mCONFIG_MFD_INTEL_PMC_BXT=mCONFIG_MFD_IQS62X=mCONFIG_MFD_JANZ_CMODIO=mCONFIG_MFD_KEMPLD=mCONFIG_MFD_88PM800=mCONFIG_MFD_88PM805=mCONFIG_MFD_88PM860X=yCONFIG_MFD_MAX14577=y
# CONFIG_MFD_MAX77541 is not setCONFIG_MFD_MAX77693=yCONFIG_MFD_MAX77843=yCONFIG_MFD_MAX8907=mCONFIG_MFD_MAX8925=yCONFIG_MFD_MAX8997=yCONFIG_MFD_MAX8998=yCONFIG_MFD_MT6360=m
# CONFIG_MFD_MT6370 is not setCONFIG_MFD_MT6397=mCONFIG_MFD_MENF21BMC=m
# CONFIG_MFD_OCELOT is not setCONFIG_EZX_PCAP=yCONFIG_MFD_VIPERBOARD=mCONFIG_MFD_RETU=mCONFIG_MFD_PCF50633=mCONFIG_PCF50633_ADC=mCONFIG_PCF50633_GPIO=m
# CONFIG_MFD_SY7636A is not setCONFIG_MFD_RDC321X=mCONFIG_MFD_RT4831=mCONFIG_MFD_RT5033=m
# CONFIG_MFD_RT5120 is not setCONFIG_MFD_RC5T583=yCONFIG_MFD_SI476X_CORE=mCONFIG_MFD_SM501=mCONFIG_MFD_SM501_GPIO=yCONFIG_MFD_SKY81452=mCONFIG_MFD_SYSCON=yCONFIG_MFD_LP3943=mCONFIG_MFD_LP8788=yCONFIG_MFD_TI_LMU=mCONFIG_MFD_PALMAS=yCONFIG_TPS6105X=mCONFIG_TPS65010=mCONFIG_TPS6507X=mCONFIG_MFD_TPS65086=mCONFIG_MFD_TPS65090=yCONFIG_MFD_TI_LP873X=mCONFIG_MFD_TPS6586X=yCONFIG_MFD_TPS65910=yCONFIG_MFD_TPS65912=yCONFIG_MFD_TPS65912_I2C=yCONFIG_MFD_TPS65912_SPI=y
# CONFIG_MFD_TPS6594_I2C is not set
# CONFIG_MFD_TPS6594_SPI is not setCONFIG_TWL4030_CORE=yCONFIG_MFD_TWL4030_AUDIO=yCONFIG_TWL6040_CORE=yCONFIG_MFD_WL1273_CORE=mCONFIG_MFD_LM3533=mCONFIG_MFD_TQMX86=mCONFIG_MFD_VX855=mCONFIG_MFD_ARIZONA=mCONFIG_MFD_ARIZONA_I2C=mCONFIG_MFD_ARIZONA_SPI=mCONFIG_MFD_CS47L24=yCONFIG_MFD_WM5102=yCONFIG_MFD_WM5110=yCONFIG_MFD_WM8997=yCONFIG_MFD_WM8998=yCONFIG_MFD_WM8400=yCONFIG_MFD_WM831X=yCONFIG_MFD_WM831X_I2C=yCONFIG_MFD_WM831X_SPI=yCONFIG_MFD_WM8350=yCONFIG_MFD_WM8350_I2C=yCONFIG_MFD_WM8994=mCONFIG_MFD_WCD934X=mCONFIG_MFD_ATC260X=mCONFIG_MFD_ATC260X_I2C=mCONFIG_RAVE_SP_CORE=m
# CONFIG_MFD_INTEL_M10_BMC_SPI is not set
# CONFIG_MFD_INTEL_M10_BMC_PMCI is not set
# end of Multifunction device drivers
5,983 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 11 lines · K Prateek Nayak
I ran into a few issues when testing the series on top of tip:sched/core at commit
4475cd8bfd9b("sched/balancing: Simplify thesg_statusbitmask and use separate ->overloaded and ->overutilized flags"). All of these splats surfaced when running unixbench with Delayed Dequeue (echoingNO_DELAY_DEQUEUEto/sys/kernel/debug/sched/featuresseems to make the system stable when running Unixbench spawn)Unixbench (https://github.com/kdlucas/byte-unixbench.git) command:
./Run spawn -c 512
That plus a hackbench loop works a treat.
Quoted 4 lines
Splats appear soon into the run. Following are the splats and their corresponding code blocks from my 3rd Generation EPYC system (2 x 64C/128T):
Seems a big box is not required. With a low fat sched config (no group sched), starting ./Run spawn -c 16 (cpus*2) along with a hackbench loop reliably blows my old i7-4790 box out of the water nearly instantly.
DUMPFILE: vmcore
CPUS: 8
DATE: Mon Apr 15 07:20:29 CEST 2024
UPTIME: 00:07:23
LOAD AVERAGE: 1632.20, 684.99, 291.84
TASKS: 1401
NODENAME: homer
RELEASE: 6.9.0.g0bbac3f-master
VERSION: #7 SMP Mon Apr 15 06:40:05 CEST 2024
MACHINE: x86_64 (3591 Mhz)
MEMORY: 16 GB
PANIC: "Oops: 0000 [#1] SMP NOPTI" (check log for details)
PID: 22664
COMMAND: "hackbench"
TASK: ffff888100acbf00 [THREAD_INFO: ffff888100acbf00]
CPU: 5
STATE: TASK_WAKING (PANIC)crash> bt -sx
PID: 22664 TASK: ffff888100acbf00 CPU: 5 COMMAND: "hackbench"
#0 [ffff88817cc17920] machine_kexec+0x156 at ffffffff810642d6 #1 [ffff88817cc17970] __crash_kexec+0xd7 at ffffffff81153147 #2 [ffff88817cc17a28] crash_kexec+0x23 at ffffffff811535f3 #3 [ffff88817cc17a38] oops_end+0xbe at ffffffff810329be #4 [ffff88817cc17a58] page_fault_oops+0x81 at ffffffff81071951 #5 [ffff88817cc17ab8] exc_page_fault+0x62 at ffffffff8194f6f2 #6 [ffff88817cc17ae0] asm_exc_page_fault+0x22 at ffffffff81a00ba2
[exception RIP: pick_task_fair+71]
RIP: ffffffff810d5b57 RSP: ffff88817cc17b90 RFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff88840ed70ec0 RCX: 00000001d7ec138c
RDX: ffffffffe7a7f400 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88840ed70ec0 R8: 0000000000000c00 R9: 000000675402f79e
R10: ffff88817cc17b30 R11: 00000000000000bb R12: ffff88840ed70f40
R13: ffffffff81f64f16 R14: ffff888100acc560 R15: ffff888100acbf00
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffff88817cc17bb0] pick_next_task_fair+0x42 at ffffffff810d92c2 #8 [ffff88817cc17be0] __schedule+0x10d at ffffffff8195936d
#9 [ffff88817cc17c50] schedule+0x1c at ffffffff81959ddc
#10 [ffff88817cc17c60] schedule_timeout+0x18c at ffffffff8195fc4c #11 [ffff88817cc17cc8] unix_stream_read_generic+0x2b7 at ffffffff81869917 #12 [ffff88817cc17da8] unix_stream_recvmsg+0x68 at ffffffff8186a2d8 #13 [ffff88817cc17de0] sock_read_iter+0x159 at ffffffff8170bd69 #14 [ffff88817cc17e70] vfs_read+0x2ce at ffffffff812f195e
#15 [ffff88817cc17ef8] ksys_read+0x40 at ffffffff812f21d0
#16 [ffff88817cc17f30] do_syscall_64+0x57 at ffffffff8194b947
#17 [ffff88817cc17f50] entry_SYSCALL_64_after_hwframe+0x76 at ffffffff81a0012b
RIP: 00007f625660871e RSP: 00007ffc75d48188 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007ffc75d48200 RCX: 00007f625660871e
RDX: 0000000000000064 RSI: 00007ffc75d48190 RDI: 0000000000000007
RBP: 00007ffc75d48260 R8: 00007ffc75d48140 R9: 00007f6256612010
R10: 00007f62565f5070 R11: 0000000000000246 R12: 0000000000000064
R13: 0000000000000000 R14: 0000000000000064 R15: 0000000000000000
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002bcrash> dis pick_task_fair+710xffffffff810d5b57 <pick_task_fair+71>: cmpb $0x0,0x4c(%rax)
crash> gdb list *pick_task_fair+710xffffffff810d5b57 is in pick_task_fair (kernel/sched/fair.c:5498).
5493 SCHED_WARN_ON(cfs_rq->next->sched_delayed);
5494 return cfs_rq->next;
5495 }
5496
5497 struct sched_entity *se = pick_eevdf(cfs_rq);
5498 if (se->sched_delayed) {
5499 dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
5500 SCHED_WARN_ON(se->sched_delayed);
5501 SCHED_WARN_ON(se->on_rq);
5502 if (sched_feat(DELAY_ZERO) && se->vlag > 0)
crash> struct -ox sched_entity
struct sched_entity {
[0x0] struct load_weight load; [0x10] struct rb_node run_node;
[0x28] u64 deadline;
[0x30] u64 min_vruntime;
[0x38] struct list_head group_node;
[0x48] unsigned int on_rq;
[0x4c] unsigned char sched_delayed;
[0x4d] unsigned char custom_slice;
[0x50] u64 exec_start;
[0x58] u64 sum_exec_runtime;
[0x60] u64 prev_sum_exec_runtime;
[0x68] u64 vruntime;
[0x70] s64 vlag;
[0x78] u64 slice;
[0x80] u64 nr_migrations;
[0xc0] struct sched_avg avg;
}
SIZE: 0x100
crash>
K Prateek Nayak
reviewerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
(+ Chen Yu, Oliver Sang)
Hello Mike,
Quoted 77 lines · Mike Galbraith
Quoted 11 lines · K Prateek Nayak
I ran into a few issues when testing the series on top of tip:sched/core at commit
4475cd8bfd9b("sched/balancing: Simplify thesg_statusbitmask and use separate ->overloaded and ->overutilized flags"). All of these splats surfaced when running unixbench with Delayed Dequeue (echoingNO_DELAY_DEQUEUEto/sys/kernel/debug/sched/featuresseems to make the system stable when running Unixbench spawn)Unixbench (https://github.com/kdlucas/byte-unixbench.git) command:
./Run spawn -c 512
That plus a hackbench loop works a treat.
Quoted 4 lines
Splats appear soon into the run. Following are the splats and their corresponding code blocks from my 3rd Generation EPYC system (2 x 64C/128T):
Seems a big box is not required. With a low fat sched config (no group sched), starting ./Run spawn -c 16 (cpus*2) along with a hackbench loop reliably blows my old i7-4790 box out of the water nearly instantly.
DUMPFILE: vmcore CPUS: 8 DATE: Mon Apr 15 07:20:29 CEST 2024 UPTIME: 00:07:23 LOAD AVERAGE: 1632.20, 684.99, 291.84 TASKS: 1401 NODENAME: homer RELEASE: 6.9.0.g0bbac3f-master VERSION: #7 SMP Mon Apr 15 06:40:05 CEST 2024 MACHINE: x86_64 (3591 Mhz) MEMORY: 16 GB PANIC: "Oops: 0000 [#1] SMP NOPTI" (check log for details) PID: 22664 COMMAND: "hackbench" TASK: ffff888100acbf00 [THREAD_INFO: ffff888100acbf00] CPU: 5 STATE: TASK_WAKING (PANIC)crash> bt -sx
PID: 22664 TASK:ffff888100acbf00CPU: 5 COMMAND: "hackbench"
#0 [ffff88817cc17920]machine_kexec+0x156atffffffff810642d6#1 [ffff88817cc17970]__crash_kexec+0xd7atffffffff81153147#2 [ffff88817cc17a28]crash_kexec+0x23atffffffff811535f3#3 [ffff88817cc17a38]oops_end+0xbeatffffffff810329be#4 [ffff88817cc17a58]page_fault_oops+0x81atffffffff81071951#5 [ffff88817cc17ab8]exc_page_fault+0x62atffffffff8194f6f2#6 [ffff88817cc17ae0]asm_exc_page_fault+0x22atffffffff81a00ba2[exception RIP: pick_task_fair+71] RIP: ffffffff810d5b57 RSP: ffff88817cc17b90 RFLAGS: 00010046 RAX: 0000000000000000 RBX: ffff88840ed70ec0 RCX: 00000001d7ec138c RDX: ffffffffe7a7f400 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffff88840ed70ec0 R8: 0000000000000c00 R9: 000000675402f79e R10: ffff88817cc17b30 R11: 00000000000000bb R12: ffff88840ed70f40 R13: ffffffff81f64f16 R14: ffff888100acc560 R15: ffff888100acbf00 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#7 [
ffff88817cc17bb0]pick_next_task_fair+0x42atffffffff810d92c2#8 [ffff88817cc17be0]__schedule+0x10datffffffff8195936d
#9 [ffff88817cc17c50]schedule+0x1catffffffff81959ddc
#10 [ffff88817cc17c60]schedule_timeout+0x18catffffffff8195fc4c#11 [ffff88817cc17cc8]unix_stream_read_generic+0x2b7atffffffff81869917#12 [ffff88817cc17da8]unix_stream_recvmsg+0x68atffffffff8186a2d8#13 [ffff88817cc17de0]sock_read_iter+0x159atffffffff8170bd69#14 [ffff88817cc17e70]vfs_read+0x2ceatffffffff812f195e
#15 [ffff88817cc17ef8]ksys_read+0x40atffffffff812f21d0
#16 [ffff88817cc17f30]do_syscall_64+0x57atffffffff8194b947
#17 [ffff88817cc17f50]entry_SYSCALL_64_after_hwframe+0x76atffffffff81a0012bRIP: 00007f625660871e RSP: 00007ffc75d48188 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 00007ffc75d48200 RCX: 00007f625660871e RDX: 0000000000000064 RSI: 00007ffc75d48190 RDI: 0000000000000007 RBP: 00007ffc75d48260 R8: 00007ffc75d48140 R9: 00007f6256612010 R10: 00007f62565f5070 R11: 0000000000000246 R12: 0000000000000064 R13: 0000000000000000 R14: 0000000000000064 R15: 0000000000000000 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Quoted 10 lines
crash> dis pick_task_fair+71
0xffffffff810d5b57<pick_task_fair+71>: cmpb $0x0,0x4c(%rax)
crash> gdb list *pick_task_fair+710xffffffff810d5b57is inpick_task_fair(kernel/sched/fair.c:5498).
5493SCHED_WARN_ON(cfs_rq->next->sched_delayed);
5494 returncfs_rq->next;
5495 }
5496
5497 structsched_entity*se =pick_eevdf(cfs_rq);
5498 if (se->sched_delayed) {
Wondering if you are running into the issue where pick_eevdf() returns NULL despite there being runnable CFS task on the runqueue. Can you try running with the following patch from Chenyu -
https://lore.kernel.org/lkml/[email protected]/
Quoted 23 lines
5499
dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
5500SCHED_WARN_ON(se->sched_delayed);
5501SCHED_WARN_ON(se->on_rq);
5502 if (sched_feat(DELAY_ZERO)&&se->vlag> 0)
crash> struct -oxsched_entitystruct sched_entity { [0x0] struct load_weight load; [0x10] struct rb_node run_node; [0x28] u64 deadline; [0x30] u64 min_vruntime; [0x38] struct list_head group_node; [0x48] unsigned int on_rq; [0x4c] unsigned char sched_delayed; [0x4d] unsigned char custom_slice; [0x50] u64 exec_start; [0x58] u64 sum_exec_runtime; [0x60] u64 prev_sum_exec_runtime; [0x68] u64 vruntime; [0x70] s64 vlag; [0x78] u64 slice; [0x80] u64 nr_migrations; [0xc0] struct sched_avg avg; }
Quoted 3 lines
SIZE:
0x100
crash>
--
Thanks and Regards,
Prateek
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 5 lines · K Prateek Nayak
Wondering if you are running into the issue where
pick_eevdf()returns NULL despite there being runnable CFS task on the runqueue. Can you try running with the following patch from Chenyu -
https://lore.kernel.org/lkml/[email protected]/
I hadn't tried it in master branch, but had in my 6.1-eevdf tree which contains the previous delay dequeue version, because I found that it too would explode, which surprised me having run it quite a bit woe free. Patch helped 6.1-eevdf not at all. I just tried it in master to be sure, and while the death throes were not identical, this one began with SCHED_WARN_ON() in put_prev_entity(), box still went belly up.
BTW/FWIW, turning on AUTOGROUP seemingly rendered box immune (unless booted with noautogroup 'course), and that immunity even held when I launched all players in the same cgroup (hmm).
-Mike
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Greetings,
I tossed a couple rocks at it today, and seem to have hit the little bugger. The root cause seems to be doing the delay dequeue business on exiting tasks. Hunk #1 of hacklet below seems to quell the explosions.
crash> bt -sx
PID: 21722 TASK: ffff88815710ee40 CPU: 6 COMMAND: "hackbench"
#0 [ffff88822773bb90] machine_kexec+0x156 at ffffffff810642d6 #1 [ffff88822773bbe0] __crash_kexec+0xd7 at ffffffff81152a07 #2 [ffff88822773bc98] crash_kexec+0x23 at ffffffff81152eb3 #3 [ffff88822773bca8] oops_end+0xbe at ffffffff810329be #4 [ffff88822773bcc8] page_fault_oops+0x81 at ffffffff81071951 #5 [ffff88822773bd28] exc_page_fault+0x62 at ffffffff8194e9e2 #6 [ffff88822773bd50] asm_exc_page_fault+0x22 at ffffffff81a00ba2
[exception RIP: pick_next_task_fair+178]
RIP: ffffffff810d8d12 RSP: ffff88822773be00 RFLAGS: 00010006
RAX: ffff88813cb780b8 RBX: ffff88840edb0e80 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88813cb78080 RDI: ffff88840ec30f00
RBP: ffff88813cb78000 R8: ffff88815710eec0 R9: 0000000000000001
R10: ffff88822773bdc8 R11: 0000000000000013 R12: 0000000000030e80
R13: ffff88815710ee40 R14: ffff88813cb78080 R15: ffff88815710ee40
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffff88822773be28] __schedule+0x10d at ffffffff8195865d
#8 [ffff88822773be98] do_task_dead+0x3e at ffffffff810cc00e
#9 [ffff88822773beb0] do_exit+0x770 at ffffffff8108f0e0
#10 [ffff88822773bf00] do_group_exit+0x2c at ffffffff8108f64c
#11 [ffff88822773bf28] __x64_sys_exit_group+0x14 at ffffffff8108f6f4 #12 [ffff88822773bf30] do_syscall_64+0x57 at ffffffff8194ac37
#13 [ffff88822773bf50] entry_SYSCALL_64_after_hwframe+0x76 at ffffffff81a0012b
RIP: 00007f4f2aa76136 RSP: 00007ffcbba84748 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4f2aa76136
RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
RBP: 00007f4f2ab86970 R8: 00000000000000e7 R9: ffffffffffffff80
R10: 0000000000000004 R11: 0000000000000246 R12: 00007f4f2ab86970
R13: 0000000000000001 R14: 00007f4f2ab8a328 R15: 0000000000000000
ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b
crash> task_struct ffff88815710ee40 | grep sched_delayed
sched_delayed = 1 '\001',crash>
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/fair.c +3 -2
@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;
@@ -5495,14 +5496,14 @@ pick_next_entity(struct rq *rq, struct c }struct sched_entity *se =pick_eevdf (cfs_rq); -if (se->sched_delayed) { +while (se && se->sched_delayed) {dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED );SCHED_WARN_ON (se->sched_delayed);SCHED_WARN_ON (se->on_rq);if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) se->vlag =0 ; -return NULL ; + se =pick_eevdf (cfs_rq); }return se; }
K Prateek Nayak
reviewerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Hello Mike,
Quoted 39 lines · Mike Galbraith
Greetings,
I tossed a couple rocks at it today, and seem to have hit the little bugger. The root cause seems to be doing the delay dequeue business on exiting tasks. Hunk #1 of hacklet below seems to quell the explosions.
crash> bt -sx
PID: 21722 TASK:ffff88815710ee40CPU: 6 COMMAND: "hackbench"
#0 [ffff88822773bb90]machine_kexec+0x156atffffffff810642d6#1 [ffff88822773bbe0]__crash_kexec+0xd7atffffffff81152a07#2 [ffff88822773bc98]crash_kexec+0x23atffffffff81152eb3#3 [ffff88822773bca8]oops_end+0xbeatffffffff810329be#4 [ffff88822773bcc8]page_fault_oops+0x81atffffffff81071951#5 [ffff88822773bd28]exc_page_fault+0x62atffffffff8194e9e2#6 [ffff88822773bd50]asm_exc_page_fault+0x22atffffffff81a00ba2[exception RIP: pick_next_task_fair+178] RIP: ffffffff810d8d12 RSP: ffff88822773be00 RFLAGS: 00010006 RAX: ffff88813cb780b8 RBX: ffff88840edb0e80 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88813cb78080 RDI: ffff88840ec30f00 RBP: ffff88813cb78000 R8: ffff88815710eec0 R9: 0000000000000001 R10: ffff88822773bdc8 R11: 0000000000000013 R12: 0000000000030e80 R13: ffff88815710ee40 R14: ffff88813cb78080 R15: ffff88815710ee40 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#7 [
ffff88822773be28]__schedule+0x10datffffffff8195865d
#8 [ffff88822773be98]do_task_dead+0x3eatffffffff810cc00e
#9 [ffff88822773beb0]do_exit+0x770atffffffff8108f0e0
#10 [ffff88822773bf00]do_group_exit+0x2catffffffff8108f64c
#11 [ffff88822773bf28]__x64_sys_exit_group+0x14atffffffff8108f6f4#12 [ffff88822773bf30]do_syscall_64+0x57atffffffff8194ac37
#13 [ffff88822773bf50]entry_SYSCALL_64_after_hwframe+0x76atffffffff81a0012bRIP: 00007f4f2aa76136 RSP: 00007ffcbba84748 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4f2aa76136 RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000 RBP: 00007f4f2ab86970 R8: 00000000000000e7 R9: ffffffffffffff80 R10: 0000000000000004 R11: 0000000000000246 R12: 00007f4f2ab86970 R13: 0000000000000001 R14: 00007f4f2ab8a328 R15: 0000000000000000 ORIG_RAX: 00000000000000e7 CS: 0033 SS: 002b crash> task_struct ffff88815710ee40 | grep sched_delayed sched_delayed = 1 '\001',
Quoted 2 lines
crash>
My machine too survived the spawn test with the below patch. Thank you for looking into this. I'll resume testing with the below patch included.
--
Thanks and Regards,
Prateek
Quoted 32 lines
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c +3
-2@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;@@ -5495,14 +5496,14 @@ pick_next_entity(struct rq *rq, struct c }struct sched_entity *se =pick_eevdf (cfs_rq); -if (se->sched_delayed) { +while (se && se->sched_delayed) {dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED );SCHED_WARN_ON (se->sched_delayed);SCHED_WARN_ON (se->on_rq);if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) se->vlag =0 ; -return NULL ; + se =pick_eevdf (cfs_rq); }return se; }
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Mike GalbraithQuoted 6 lines · Mike Galbraith
Greetings,
I tossed a couple rocks at it today, and seem to have hit the little bugger. The root cause seems to be doing the delay dequeue business on exiting tasks. Hunk #1 of hacklet below seems to quell the explosions.
Neat, Thanks! I was just about ready to go look at this again.
Quoted 33 lines
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c +3
-2@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;@@ -5495,14 +5496,14 @@ pick_next_entity(struct rq *rq, struct c }struct sched_entity *se =pick_eevdf (cfs_rq); -if (se->sched_delayed) { +while (se && se->sched_delayed) {dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED );SCHED_WARN_ON (se->sched_delayed);SCHED_WARN_ON (se->on_rq);if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) se->vlag =0 ; -return NULL ; + se =pick_eevdf (cfs_rq); }return se; }
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Mike GalbraithQuoted 10 lines · Mike Galbraith
The root cause seems to be doing the delay dequeue business on exiting tasks.
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5380
Quoted 7 lines
kernel/sched/fair.c +1
-0@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;
So I think this can be easier done in dequeue_task_fair(), where we still know this is a task.
Perhaps something like (I'll test later):
if (p->flags & PF_EXITING) flags &= ~DEQUEUE_SLEEP;
But now I need to go think about the case of removing a cgroup... *urgh*.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 22 lines · Peter Zijlstra
Quoted 10 lines · Mike Galbraith
The root cause seems to be doing the delay dequeue business on exiting tasks.
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5380
Quoted 7 lines
kernel/sched/fair.c +1
-0@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;So I think this can be easier done in
dequeue_task_fair(), where we still know this is a task.
Quoted 7 lines
Perhaps something like (I'll test later):
if (p->flags & PF_EXITING) flags &= ~DEQUEUE_SLEEP;But now I need to go think about the case of removing a cgroup... *urgh*.
I ended up with the below instead; lemme go run this unixbench spawn on it.
---
kernel/sched/fair.c +32 -13
@@ -8429,7 +8431,20 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)static void task_dead_fair (struct task_struct *p) { -remove_entity_load_avg (&p->se); +struct sched_entity *se = &p->se; + +if (p->se.sched_delayed) { +struct rq_flags rf; +struct rq *rq; + + rq =task_rq_lock (p, &rf); +update_rq_clock (rq); +if (se->sched_delayed) +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +task_rq_unlock (rq, p, &rf); + } + +remove_entity_load_avg (se); }/*
@@ -13089,28 +13104,34 @@ void online_fair_sched_group(struct task_group *tg)void unregister_fair_sched_group (struct task_group *tg) { -unsigned long flags; -struct rq *rq;int cpu;destroy_cfs_bandwidth (tg_cfs_bandwidth (tg));for_each_possible_cpu (cpu) { -if (tg->se[cpu]) -remove_entity_load_avg (tg->se[cpu]); +struct cfs_rq *cfs_rq = tg->cfs_rq[cpu]; +struct sched_entity *se = tg->se[cpu]; +struct rq *rq =cpu_rq (cpu); + +if (se) { +if (se->sched_delayed) { +guard (rq_lock_irqsave)(rq); +update_rq_clock (rq); +if (se->sched_delayed) +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +list_del_leaf_cfs_rq (cfs_rq); + } +remove_entity_load_avg (se); + }/* * Only empty task groups can be destroyed; so we can speculatively * check on_list without danger of it being re-added. */ -if (!tg->cfs_rq[cpu]->on_list) -continue ; - - rq =cpu_rq (cpu); - -raw_spin_rq_lock_irqsave (rq, flags); -list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_rq_unlock_irqrestore (rq, flags); +if (cfs_rq->on_list) { +guard (rq_lock_irqsave)(rq); +list_del_leaf_cfs_rq (cfs_rq); + } } }
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 10 lines · Mike Galbraith
The root cause seems to be doing the delay dequeue business on exiting tasks.
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5380
Quoted 7 lines
kernel/sched/fair.c +1
-0@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && sleep && + !(entity_is_task (se) && (task_of (se)->flags &PF_EXITING )) && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;So I think this can be easier done in
dequeue_task_fair(), where we still know this is a task.Perhaps something like (I'll test later):
Quoted 7 lines
Quoted 5 lines
if (p->flags & PF_EXITING) flags &= ~DEQUEUE_SLEEP;But now I need to go think about the case of removing a cgroup... *urgh*.
I ended up with the below instead; lemme go run this unixbench spawn on it.
Seems to survive that.
I pushed out the patches with updates to queue/sched/eevdf
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I ended up with the below instead; lemme go run this unixbench spawn on it.
Seems to survive that.
I pushed out the patches with updates to queue/sched/eevdf
Yup, solid... but fwiw, tbench liked the previous version better.
trusty (with a 'c') ole i7-4790 box, tbench 8
for i in 1 2 3; do tbench.sh 8 10 2>&1|grep Throughput; done
6.9.0.gc942a0c-master +eevdf.currentNO_DELAY_DEQUEUE
Throughput 3285.04 MB/sec 8 clients 8 procs max_latency=3.481 ms Throughput 3289.66 MB/sec 8 clients 8 procs max_latency=8.124 ms Throughput 3293.83 MB/sec 8 clients 8 procs max_latency=2.210 ms DELAY_DEQUEUE
Throughput 3246.3 MB/sec 8 clients 8 procs max_latency=2.181 ms Throughput 3236.96 MB/sec 8 clients 8 procs max_latency=6.988 ms Throughput 3248.6 MB/sec 8 clients 8 procs max_latency=2.130 ms
6.9.0.gc942a0c-master +eevdf.prevNO_DELAY_DEQUEUE
Throughput 3457.92 MB/sec 8 clients 8 procs max_latency=3.885 ms Throughput 3470.95 MB/sec 8 clients 8 procs max_latency=4.475 ms Throughput 3467.87 MB/sec 8 clients 8 procs max_latency=2.182 ms DELAY_DEQUEUE
Throughput 3712.96 MB/sec 8 clients 8 procs max_latency=4.231 ms Throughput 3667.87 MB/sec 8 clients 8 procs max_latency=5.020 ms Throughput 3679.65 MB/sec 8 clients 8 procs max_latency=2.847 ms
Trees are identical modulo extracted eevdf additions. The previous win that put eevdf on par with cfs went missing.. and then some.
For reference, cfs vs eevdf log extract for previously mentioned gain.
6.1.87-cfs
Throughput 3660.98 MB/sec 8 clients 8 procs max_latency=2.204 ms
Throughput 3678.67 MB/sec 8 clients 8 procs max_latency=10.127 ms
Throughput 3631.89 MB/sec 8 clients 8 procs max_latency=13.019 ms
1.0006.1.87-eevdf - naked eevdf +fixes
Throughput 3441.86 MB/sec 8 clients 8 procs max_latency=3.943 ms Throughput 3439.68 MB/sec 8 clients 8 procs max_latency=4.285 ms Throughput 3432.28 MB/sec 8 clients 8 procs max_latency=3.557 ms
vs cfs .940
6.1.87-eevdf +delay_dequeue.prev patch setDELAY_DEQUEUE
Throughput 3696.94 MB/sec 8 clients 8 procs max_latency=2.179 ms Throughput 3694.64 MB/sec 8 clients 8 procs max_latency=6.322 ms Throughput 3654.49 MB/sec 8 clients 8 procs max_latency=4.101 ms
vs cfs 1.006
box waxes nostalgic (son, when I was yo age [flex]) 4.19.312
Throughput 4099.07 MB/sec 8 clients 8 procs max_latency=2.169 ms Throughput 4107.49 MB/sec 8 clients 8 procs max_latency=12.404 ms Throughput 4118.41 MB/sec 8 clients 8 procs max_latency=14.150 ms
-Mike
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 1 line · Mike Galbraith
fwiw, tbench liked the previous version better.
The culprit lies somewhere in the new PREEMPT_SHORT patch.
-Mike
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 4 lines · Mike Galbraith
Quoted 1 line · Mike Galbraith
fwiw, tbench liked the previous version better.
The culprit lies somewhere in the new
PREEMPT_SHORTpatch.
<squint>
Ah, moving the curr eligibility check into pick_curr() left an ineligible curr to be seen/used further down in pick_eevdf().
found:
if (!best || (curr && entity_before(curr, best)))
best = curr;Nit: PREEMPT_SHORT depending on RUN_TO_PARITY looks odd.
-Mike
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 6 lines · Mike Galbraith
Quoted 4 lines · Mike Galbraith
Quoted 1 line · Mike Galbraith
fwiw, tbench liked the previous version better.
The culprit lies somewhere in the new
PREEMPT_SHORTpatch.
Quoted 8 lines
<squint>
Ah, moving the curr eligibility check into
pick_curr()left an ineligible curr to be seen/used further down inpick_eevdf().found:
if (!best || (curr && entity_before(curr, best))) best = curr;
Hmm yes, over aggressive cleanup that. Let me try again.
Quoted 1 line
Nit:
PREEMPT_SHORTdepending onRUN_TO_PARITYlooks odd.
The thinking was that without RUN_TO_PARITY we'll always do a full pick and it will always pick a (new) shorter deadline task over current.
The PREEMPRT_SHORT thing really is an exception to avoid RUN_TO_PARITY from ruining latency game for short tasks.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHi Peter,
Quoted 19 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Quoted 6 lines
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
XXX test the cfs-throttle stuff
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Playing around with a Pixel 6 running a 6.6-based kernel with this series backported on top, I spotted a noticeable performance improvement on the speedometer benchmark:
- m6.6-stock-* is the 6.6 mainline Android kernel unmodified.
- m6.6-eevdf-complete-* is the 6.6 mainline Android kernel with this series applied on top (along with a few required backported patches).
+-------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-------------------+-----------------------+-----------+ | Speedometer Score | m6.6-stock-1 | 0.0% | | Speedometer Score | m6.6-stock-2 | 1.23% | | Speedometer Score | m6.6-stock-3 | -0.22% | | Speedometer Score | m6.6-eevdf-complete-1 | 4.54% | | Speedometer Score | m6.6-eevdf-complete-2 | 4.95% | | Speedometer Score | m6.6-eevdf-complete-3 | 6.07% | +-------------------+-----------------------+-----------+
Also some interesting improvements in terms of frame timing for the uibenchjanktests benchmark. In particular the metrics of missed deadlines and jank (late) frames, which seems to indicate better latencies.
+-----------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-----------------------+-----------------------+-----------+ | gfx-avg-frame-time-50 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-90 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-95 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-99 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-50 | m6.6-stock-2 | 3.46 | | gfx-avg-frame-time-90 | m6.6-stock-2 | 1.19 | | gfx-avg-frame-time-95 | m6.6-stock-2 | 0.24 | | gfx-avg-frame-time-99 | m6.6-stock-2 | 0.48 | | gfx-avg-frame-time-50 | m6.6-eevdf-complete-1 | -30.45 | | gfx-avg-frame-time-90 | m6.6-eevdf-complete-1 | -48.44 | | gfx-avg-frame-time-95 | m6.6-eevdf-complete-1 | -51.32 | | gfx-avg-frame-time-99 | m6.6-eevdf-complete-1 | -52.48 | | gfx-avg-frame-time-50 | m6.6-eevdf-complete-2 | -30.32 | | gfx-avg-frame-time-90 | m6.6-eevdf-complete-2 | -48.16 | | gfx-avg-frame-time-95 | m6.6-eevdf-complete-2 | -51.08 | | gfx-avg-frame-time-99 | m6.6-eevdf-complete-2 | -51.7 | +-----------------------+-----------------------+-----------+
+-----------------------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-----------------------------------+-----------------------+-----------+ | gfx-avg-num-frame-deadline-missed | m6.6-stock-1 | 0.0 | | gfx-max-num-frame-deadline-missed | m6.6-stock-1 | 0.0 | | gfx-avg-num-frame-deadline-missed | m6.6-stock-2 | -3.21 | | gfx-max-num-frame-deadline-missed | m6.6-stock-2 | -3.21 | | gfx-avg-num-frame-deadline-missed | m6.6-eevdf-complete-1 | -85.29 | | gfx-max-num-frame-deadline-missed | m6.6-eevdf-complete-1 | -85.29 | | gfx-avg-num-frame-deadline-missed | m6.6-eevdf-complete-2 | -84.8 | | gfx-max-num-frame-deadline-missed | m6.6-eevdf-complete-2 | -84.8 | +-----------------------------------+-----------------------+-----------+
+----------------------------+-----------------------+-----------+ | metric | tag | perc_diff | +----------------------------+-----------------------+-----------+ | gfx-avg-high-input-latency | m6.6-stock-1 | 0.0 | | gfx-max-high-input-latency | m6.6-stock-1 | 0.0 | | gfx-avg-high-input-latency | m6.6-stock-2 | 0.93 | | gfx-max-high-input-latency | m6.6-stock-2 | 0.93 | | gfx-avg-high-input-latency | m6.6-eevdf-complete-1 | -18.35 | | gfx-max-high-input-latency | m6.6-eevdf-complete-1 | -18.35 | | gfx-avg-high-input-latency | m6.6-eevdf-complete-2 | -18.05 | | gfx-max-high-input-latency | m6.6-eevdf-complete-2 | -18.05 | +----------------------------+-----------------------+-----------+
+--------------+-----------------------+-----------+ | metric | tag | perc_diff | +--------------+-----------------------+-----------+ | gfx-avg-jank | m6.6-stock-1 | 0.0 | | gfx-max-jank | m6.6-stock-1 | 0.0 | | gfx-avg-jank | m6.6-stock-2 | 1.56 | | gfx-max-jank | m6.6-stock-2 | 1.56 | | gfx-avg-jank | m6.6-eevdf-complete-1 | -82.81 | | gfx-max-jank | m6.6-eevdf-complete-1 | -82.81 | | gfx-avg-jank | m6.6-eevdf-complete-2 | -78.12 | | gfx-max-jank | m6.6-eevdf-complete-2 | -78.12 | +--------------+-----------------------+-----------+
Bisecting through the patches in this series, I ended up with patch 08/10 as the one that improved things overall for these benchmarks.
I'd like to investigate this further to understand the reason behind some of these dramatic improvements.
Quoted 417 lines
--- include/linux/sched.h | 1 kernel/sched/core.c | 22 +++++-- kernel/sched/fair.c | 148 +++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/features.h | 12 +++ kernel/sched/sched.h | 2 5 files changed, 167 insertions(+), 18 deletions(-)include/linux/sched.h +1
-0@@ -542,6 +542,7 @@ struct sched_entity {struct list_head group_node;unsigned int on_rq; +unsigned int sched_delayed;u64 exec_start;u64 sum_exec_runtime;kernel/sched/core.c +18
-4@@ -2154,10 +2154,14 @@ void activate_task(struct rq *rq, structvoid deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -WRITE_ONCE (p->on_rq, (flags &DEQUEUE_SLEEP ) ?0 :TASK_ON_RQ_MIGRATING ); -ASSERT_EXCLUSIVE_WRITER (p->on_rq); +bool sleep = flags &DEQUEUE_SLEEP ; -dequeue_task (rq, p, flags); +if (dequeue_task (rq, p, flags)) { +WRITE_ONCE (p->on_rq, sleep ?0 :TASK_ON_RQ_MIGRATING ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); + }else { +SCHED_WARN_ON (!sleep);/* only sleep can fail */ + } }static inline int __normal_prio (int policy,int rt_prio,int nice)@@ -3858,12 +3862,17 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) { +update_rq_clock (rq); +if (p->se.sched_delayed) { +/* mustn't run a delayed task */ +SCHED_WARN_ON (task_on_cpu (rq, p)); +enqueue_task (rq, p,ENQUEUE_DELAYED ); + }if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -update_rq_clock (rq);wakeup_preempt (rq, p, wake_flags); }ttwu_do_wakeup (p);@@ -4243,11 +4252,16 @@ int try_to_wake_up(struct task_struct *p* case the whole 'p->on_rq && ttwu_runnable()' case below * without taking any locks. * +* Specifically, given current runs ttwu() we must be before +* schedule()'s deactivate_task(), as such this must not +* observe sched_delayed. +* * In particular: * - we rely on Program-Order guarantees for all the ordering, * - we're serialized against set_special_state() by virtue of * it disabling IRQs (this allows not taking ->pi_lock). */ +SCHED_WARN_ON (p->se.sched_delayed);if (!ttwu_state_match (p, state, &success)) goto out;kernel/sched/fair.c +134
-14@@ -5270,6 +5270,10 @@ static inline int cfs_rq_throttled(strucstatic inline bool cfs_bandwidth_used (void );static void +requeue_delayed_entity (struct sched_entity *se); + +/* XXX bool and pull in the requeue_delayed_entity thing */ +static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {bool curr = cfs_rq->curr == se;@@ -5356,18 +5360,33 @@ static void clear_buddies(struct cfs_rq static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); -static void +static bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -int action =UPDATE_TG ; - -if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) - action |=DO_DETACH ; -/* * Update run-time statistics of the 'current'. */ -update_curr (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { +SCHED_WARN_ON (!se->sched_delayed); + se->sched_delayed =0 ; + }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); + +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) { +if (cfs_rq->next == se) + cfs_rq->next =NULL ; + se->sched_delayed =1 ; +return false; + } + } + +int action =UPDATE_TG ; +if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) + action |=DO_DETACH ;/* * When dequeuing a sched_entity, we must: @@ -5407,6 +5426,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq); + +return true; } static void@@ -5432,6 +5453,7 @@ set_next_entity(struct cfs_rq *cfs_rq, s }update_stats_curr_start (cfs_rq, se); +SCHED_WARN_ON (cfs_rq->curr); cfs_rq->curr = se;/* @@ -5452,6 +5474,8 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +/* * Pick the next process, keeping these things in mind, in this order: * 1) keep things fair between processes/task groups @@ -5460,16 +5484,29 @@ set_next_entity(struct cfs_rq *cfs_rq, s* 4) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity (struct cfs_rq *cfs_rq) +pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq) {/* * Enabling NEXT_BUDDY will affect latency but not fairness. */ if (sched_feat (NEXT_BUDDY ) && - cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) + cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) { +/* ->next will never be delayed */ +SCHED_WARN_ON (cfs_rq->next->sched_delayed);return cfs_rq->next; + } -return pick_eevdf (cfs_rq); +struct sched_entity *se =pick_eevdf (cfs_rq); +if (se->sched_delayed) { +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +SCHED_WARN_ON (se->sched_delayed); +SCHED_WARN_ON (se->on_rq); +if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) + se->vlag =0 ; + +return NULL ; + } +return se; }static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq);@@ -5493,6 +5530,7 @@ static void put_prev_entity(struct cfs_r/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } +SCHED_WARN_ON (cfs_rq->curr != prev); cfs_rq->curr =NULL ; }@@ -5793,6 +5831,10 @@ static bool throttle_cfs_rq(struct cfs_rif (!se->on_rq) goto done; +/* +* XXX should be fine vs sched_delay; if won't run after this. +* Either pick dequeues it, or unthrottle. Double check!! +*/ dequeue_entity (qcfs_rq, se,DEQUEUE_SLEEP );if (cfs_rq_is_idle (group_cfs_rq (se)))@@ -5882,8 +5924,11 @@ void unthrottle_cfs_rq(struct cfs_rq *cffor_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); -if (se->on_rq) +if (se->on_rq) { +if (se->sched_delayed) +requeue_delayed_entity (se);break ; + }enqueue_entity (qcfs_rq, se,ENQUEUE_WAKEUP );if (cfs_rq_is_idle (group_cfs_rq (se)))@@ -6729,6 +6774,40 @@ static int sched_idle_cpu(int cpu) }#endif +static void +requeue_delayed_entity (struct sched_entity *se) +{ +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +/* +* se->sched_delayed should imply both: se->on_rq == 1 and +* cfs_rq->curr != se. Because a delayed entity is one that is still on +* the runqueue competing until elegibility. +* +* Except for groups, consider current going idle and newidle pulling a +* task in the same group -- in that case 'cfs_rq->curr == se'. +*/ +SCHED_WARN_ON (!se->sched_delayed); +SCHED_WARN_ON (!se->on_rq); +SCHED_WARN_ON (entity_is_task (se) && cfs_rq->curr == se); + +if (sched_feat (DELAY_ZERO )) { +update_entity_lag (cfs_rq, se); +if (se->vlag >0 ) { + cfs_rq->nr_running--; +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); + se->vlag =0 ; +place_entity (cfs_rq, se,0 ); +if (se != cfs_rq->curr) +__enqueue_entity (cfs_rq, se); + cfs_rq->nr_running++; + } + } + + se->sched_delayed =0 ; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -6742,6 +6821,11 @@ enqueue_task_fair(struct rq *rq, structint idle_h_nr_running =task_has_idle_policy (p);int task_new = !(flags &ENQUEUE_WAKEUP ); +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + } +/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. @@ -6759,8 +6843,11 @@ enqueue_task_fair(struct rq *rq, structcpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT );for_each_sched_entity (se) { -if (se->on_rq) +if (se->on_rq) { +if (se->sched_delayed) +requeue_delayed_entity (se);break ; + } cfs_rq =cfs_rq_of (se);enqueue_entity (cfs_rq, se, flags);@@ -6836,6 +6923,7 @@ static int dequeue_entities(struct rq *r {bool was_sched_idle =sched_idle_rq (rq);bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ;struct task_struct *p =NULL ;struct cfs_rq *cfs_rq;int idle_h_nr_running;@@ -6849,7 +6937,13 @@ static int dequeue_entities(struct rq *rfor_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); -dequeue_entity (cfs_rq, se, flags); + +if (!dequeue_entity (cfs_rq, se, flags)) { +if (p && &p->se == se) +return -1 ; + +break ; + }/* h_nr_running is the hierachical count of tasks */ if (p) {@@ -6877,6 +6971,7 @@ static int dequeue_entities(struct rq *rbreak ; } flags |=DEQUEUE_SLEEP ; + flags &= ~DEQUEUE_DELAYED ; }for_each_sched_entity (se) {@@ -6906,6 +7001,18 @@ static int dequeue_entities(struct rq *r/* balance early to pull high priority tasks */ if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; + +if (task_delayed) { +SCHED_WARN_ON (!task_sleep); +SCHED_WARN_ON (p->on_rq !=1 ); + +/* Fix-up what dequeue_task_fair() skipped */ +util_est_update (&rq->cfs, p, task_sleep); +hrtick_update (rq); + +/* Fix-up what deactivate_task() skipped. */ +WRITE_ONCE (p->on_rq,0 ); + } }return 1 ;@@ -6923,6 +7030,10 @@ static bool dequeue_task_fair(struct rqif (dequeue_entities (rq, &p->se, flags) <0 )return false; +/* +* It doesn't make sense to update util_est for the delayed dequeue +* case where ttwu will make it appear the sleep never happened. +*/ util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);return true;@@ -8463,7 +8574,9 @@ static struct task_struct *pick_task_faiif (unlikely (check_cfs_rq_runtime (cfs_rq))) goto again; - se =pick_next_entity (cfs_rq); + se =pick_next_entity (rq, cfs_rq); +if (!se) + goto again; cfs_rq =group_cfs_rq (se); }while (cfs_rq);@@ -12803,10 +12916,17 @@ static void attach_task_cfs_rq(struct tastatic void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +// XXX think harder on this +// this could actually be handled correctly I suppose; keep the whole +// se enqueued while boosted. horrible complexity though + p->se.sched_delayed =0 ; +// XXX also vlag ?!? }static void switched_to_fair (struct rq *rq,struct task_struct *p) { +SCHED_WARN_ON (p->se.sched_delayed); +attach_task_cfs_rq (p);set_task_max_allowed_capacity (p);kernel/sched/features.h +12
-0@@ -29,6 +29,18 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +* +* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. +* +* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. + */ +SCHED_FEAT(DELAY_DEQUEUE, true) +SCHED_FEAT (DELAY_ZERO , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)kernel/sched/sched.h +2
-0@@ -2245,6 +2245,7 @@ extern const u32 sched_prio_to_wmult[40#define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ #define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ +#define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */ #define ENQUEUE_WAKEUP 0x01#define ENQUEUE_RESTORE 0x02@@ -2260,6 +2261,7 @@ extern const u32 sched_prio_to_wmult[40#endif #define ENQUEUE_INITIAL 0x80#define ENQUEUE_MIGRATING 0x100 +#define ENQUEUE_DELAYED 0x200#define RETRY_TASK ((void *)-1UL)
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Hi,
Quoted 115 lines · Luis Machado
Hi Peter,
Quoted 19 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Quoted 6 lines
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
XXX test the cfs-throttle stuff
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Playing around with a Pixel 6 running a 6.6-based kernel with this series backported on top, I spotted a noticeable performance improvement on the speedometer benchmark:
- m6.6-stock-* is the 6.6 mainline Android kernel unmodified.
- m6.6-eevdf-complete-* is the 6.6 mainline Android kernel with this series applied on top (along with a few required backported patches).
+-------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-------------------+-----------------------+-----------+ | Speedometer Score | m6.6-stock-1 | 0.0% | | Speedometer Score | m6.6-stock-2 | 1.23% | | Speedometer Score | m6.6-stock-3 | -0.22% | | Speedometer Score | m6.6-eevdf-complete-1 | 4.54% | | Speedometer Score | m6.6-eevdf-complete-2 | 4.95% | | Speedometer Score | m6.6-eevdf-complete-3 | 6.07% | +-------------------+-----------------------+-----------+Also some interesting improvements in terms of frame timing for the uibenchjanktests benchmark. In particular the metrics of missed deadlines and jank (late) frames, which seems to indicate better latencies.
+-----------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-----------------------+-----------------------+-----------+ | gfx-avg-frame-time-50 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-90 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-95 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-99 | m6.6-stock-1 | 0.0 | | gfx-avg-frame-time-50 | m6.6-stock-2 | 3.46 | | gfx-avg-frame-time-90 | m6.6-stock-2 | 1.19 | | gfx-avg-frame-time-95 | m6.6-stock-2 | 0.24 | | gfx-avg-frame-time-99 | m6.6-stock-2 | 0.48 | | gfx-avg-frame-time-50 | m6.6-eevdf-complete-1 | -30.45 | | gfx-avg-frame-time-90 | m6.6-eevdf-complete-1 | -48.44 | | gfx-avg-frame-time-95 | m6.6-eevdf-complete-1 | -51.32 | | gfx-avg-frame-time-99 | m6.6-eevdf-complete-1 | -52.48 | | gfx-avg-frame-time-50 | m6.6-eevdf-complete-2 | -30.32 | | gfx-avg-frame-time-90 | m6.6-eevdf-complete-2 | -48.16 | | gfx-avg-frame-time-95 | m6.6-eevdf-complete-2 | -51.08 | | gfx-avg-frame-time-99 | m6.6-eevdf-complete-2 | -51.7 | +-----------------------+-----------------------+-----------++-----------------------------------+-----------------------+-----------+ | metric | tag | perc_diff | +-----------------------------------+-----------------------+-----------+ | gfx-avg-num-frame-deadline-missed | m6.6-stock-1 | 0.0 | | gfx-max-num-frame-deadline-missed | m6.6-stock-1 | 0.0 | | gfx-avg-num-frame-deadline-missed | m6.6-stock-2 | -3.21 | | gfx-max-num-frame-deadline-missed | m6.6-stock-2 | -3.21 | | gfx-avg-num-frame-deadline-missed | m6.6-eevdf-complete-1 | -85.29 | | gfx-max-num-frame-deadline-missed | m6.6-eevdf-complete-1 | -85.29 | | gfx-avg-num-frame-deadline-missed | m6.6-eevdf-complete-2 | -84.8 | | gfx-max-num-frame-deadline-missed | m6.6-eevdf-complete-2 | -84.8 | +-----------------------------------+-----------------------+-----------++----------------------------+-----------------------+-----------+ | metric | tag | perc_diff | +----------------------------+-----------------------+-----------+ | gfx-avg-high-input-latency | m6.6-stock-1 | 0.0 | | gfx-max-high-input-latency | m6.6-stock-1 | 0.0 | | gfx-avg-high-input-latency | m6.6-stock-2 | 0.93 | | gfx-max-high-input-latency | m6.6-stock-2 | 0.93 | | gfx-avg-high-input-latency | m6.6-eevdf-complete-1 | -18.35 | | gfx-max-high-input-latency | m6.6-eevdf-complete-1 | -18.35 | | gfx-avg-high-input-latency | m6.6-eevdf-complete-2 | -18.05 | | gfx-max-high-input-latency | m6.6-eevdf-complete-2 | -18.05 | +----------------------------+-----------------------+-----------++--------------+-----------------------+-----------+ | metric | tag | perc_diff | +--------------+-----------------------+-----------+ | gfx-avg-jank | m6.6-stock-1 | 0.0 | | gfx-max-jank | m6.6-stock-1 | 0.0 | | gfx-avg-jank | m6.6-stock-2 | 1.56 | | gfx-max-jank | m6.6-stock-2 | 1.56 | | gfx-avg-jank | m6.6-eevdf-complete-1 | -82.81 | | gfx-max-jank | m6.6-eevdf-complete-1 | -82.81 | | gfx-avg-jank | m6.6-eevdf-complete-2 | -78.12 | | gfx-max-jank | m6.6-eevdf-complete-2 | -78.12 | +--------------+-----------------------+-----------+
Quoted 6 lines
Bisecting through the patches in this series, I ended up with patch 08/10 as the one that improved things overall for these benchmarks.
I'd like to investigate this further to understand the reason behind some of these dramatic improvements.
Investigating these improvements a bit more, I noticed they came with a significantly higher power usage on the Pixel6 (where EAS is enabled). I bisected it down to the delayed dequeue patch. Disabling DELAY_DEQUEUE and DELAY_ZERO at runtime doesn't help in bringing the power usage down.
Though I don't fully understand the reason behind this change in behavior yet, I did spot the benchmark processes running almost entirely on the big core cluster, with little to no use of the little core and mid core clusters.
That would explain higher power usage and also the significant jump in performance.
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 11 lines · Luis Machado
Quoted 6 lines
Bisecting through the patches in this series, I ended up with patch 08/10 as the one that improved things overall for these benchmarks.
I'd like to investigate this further to understand the reason behind some of these dramatic improvements.
Investigating these improvements a bit more, I noticed they came with a significantly higher power usage on the Pixel6 (where EAS is enabled). I bisected it down to the delayed dequeue patch. Disabling
DELAY_DEQUEUEandDELAY_ZEROat runtime doesn't help in bringing the power usage down.
Hmm, that is unexpected. The intent was for NO_DELAY_DEQUEUE to fully disable things. I'll go have a prod at it.
Quoted 5 lines
Though I don't fully understand the reason behind this change in behavior yet, I did spot the benchmark processes running almost entirely on the big core cluster, with little to no use of the little core and mid core clusters.
That would explain higher power usage and also the significant jump in performance.
ISTR you (arm) has these tools to trace and plot the varioud util values. This should be readily reflected there if that is the case, no?
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably util_est_update() gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.
I'll go over all that again.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.
The PELT runnable values will be inflated because of delayed dequeue. cpu_util() uses those in the @boost case, and as such this can indeed affect things.
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.
Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
---
kernel/sched/fair.c +2 -0
@@ -5391,6 +5391,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );return false; } }
@@ -6817,6 +6818,7 @@ requeue_delayed_entity(struct sched_entity *se) } se->sched_delayed =0 ; +update_load_avg (qcfs_rq, se,0 ); }/*
kernel/sched/sched.h +6 -0
@@ -810,6 +810,9 @@ static inline void se_update_runnable(struct sched_entity *se)static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +if (entity_is_task (se))return !!se->on_rq;else
@@ -823,6 +826,9 @@ static inline void se_update_runnable(struct sched_entity *se) {}static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }#endif
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 38 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
---
kernel/sched/fair.c +1
-0@@ -5391,6 +5391,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );return false; } }
kernel/sched/fair.c requeue_delayed_entity(struct sched_entity *se) · line 6821
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -6817,6 +6818,7 @@ requeue_delayed_entity(struct sched_entity *se) } se->sched_delayed =0 ; +update_load_avg (qcfs_rq, se,0 );
Compiler demands this ^^^^^ be cfs_rq instead ;-) brown_paper_bags--;
Quoted 27 lines
kernel/sched/fair.c +0
-0}/* kernel/sched/sched.h +6
-0@@ -810,6 +810,9 @@ static inline void se_update_runnable(struct sched_entity *se)static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +if (entity_is_task (se))return !!se->on_rq;else @@ -823,6 +826,9 @@ static inline void se_update_runnable(struct sched_entity *se) {}static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }#endif
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.
Quoted 8 lines
Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
Thanks Peter. Let me give it a try and I'll report back.
Quoted 46 lines
---
kernel/sched/fair.c +2
-0@@ -5391,6 +5391,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );return false; } }@@ -6817,6 +6818,7 @@ requeue_delayed_entity(struct sched_entity *se) } se->sched_delayed =0 ; +update_load_avg (qcfs_rq, se,0 ); }/* kernel/sched/sched.h +6
-0@@ -810,6 +810,9 @@ static inline void se_update_runnable(struct sched_entity *se)static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +if (entity_is_task (se))return !!se->on_rq;else @@ -823,6 +826,9 @@ static inline void se_update_runnable(struct sched_entity *se) {}static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }#endif
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHi Peter,
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.
Quoted 8 lines
Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
I gave this a try on the Pixel 6, and I noticed some improvement (see below), but not enough to bring it back to the original levels.
(1) m6.6-stock - Basic EEVDF with wakeup preemption fix (baseline)
(2) m6.6-eevdf-complete: m6.6-stock plus this series.
(3) m6.6-eevdf-complete-no-delay-dequeue: (2) + NO_DELAY_DEQUEUE
(4) m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero: (2) + NO_DELAY_DEQUEUE + NO_DELAY_ZERO
(5) m6.6-eevdf-complete-no-delay-zero: (2) + NO_DELAY_ZERO
(6) m6.6-eevdf-complete-pelt-fix: (2) + the proposed load_avg update patch.
I included (3), (4) and (5) to exercise the impact of disabling the individual scheduler features.
Energy use.
+------------+------------------------------------------------------+-----------+ | cluster | tag | perc_diff | +------------+------------------------------------------------------+-----------+ | CPU | m6.6-stock | 0.0% | | CPU-Big | m6.6-stock | 0.0% | | CPU-Little | m6.6-stock | 0.0% | | CPU-Mid | m6.6-stock | 0.0% | | GPU | m6.6-stock | 0.0% | | Total | m6.6-stock | 0.0% | | CPU | m6.6-eevdf-complete | 114.51% | | CPU-Big | m6.6-eevdf-complete | 90.75% | | CPU-Little | m6.6-eevdf-complete | 98.74% | | CPU-Mid | m6.6-eevdf-complete | 213.9% | | GPU | m6.6-eevdf-complete | -7.04% | | Total | m6.6-eevdf-complete | 100.92% | | CPU | m6.6-eevdf-complete-no-delay-dequeue | 117.77% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue | 113.79% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue | 97.47% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue | 189.0% | | GPU | m6.6-eevdf-complete-no-delay-dequeue | -6.74% | | Total | m6.6-eevdf-complete-no-delay-dequeue | 103.84% | | CPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 120.45% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 113.65% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 99.04% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 201.14% | | GPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | -5.37% | | Total | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 106.38% | | CPU | m6.6-eevdf-complete-no-delay-zero | 119.05% | | CPU-Big | m6.6-eevdf-complete-no-delay-zero | 107.55% | | CPU-Little | m6.6-eevdf-complete-no-delay-zero | 98.66% | | CPU-Mid | m6.6-eevdf-complete-no-delay-zero | 206.58% | | GPU | m6.6-eevdf-complete-no-delay-zero | -5.25% | | Total | m6.6-eevdf-complete-no-delay-zero | 105.14% | | CPU | m6.6-eevdf-complete-pelt-fix | 105.56% | | CPU-Big | m6.6-eevdf-complete-pelt-fix | 100.45% | | CPU-Little | m6.6-eevdf-complete-pelt-fix | 94.4% | | CPU-Mid | m6.6-eevdf-complete-pelt-fix | 150.94% | | GPU | m6.6-eevdf-complete-pelt-fix | -3.96% | | Total | m6.6-eevdf-complete-pelt-fix | 93.31% | +------------+------------------------------------------------------+-----------+
Utilization and load levels.
+---------+------------------------------------------------------+----------+-----------+ | cluster | tag | variable | perc_diff | +---------+------------------------------------------------------+----------+-----------+ | little | m6.6-stock | load | 0.0% | | little | m6.6-stock | util | 0.0% | | little | m6.6-eevdf-complete | load | 29.56% | | little | m6.6-eevdf-complete | util | 55.4% | | little | m6.6-eevdf-complete-no-delay-dequeue | load | 42.89% | | little | m6.6-eevdf-complete-no-delay-dequeue | util | 69.47% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 51.05% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 76.55% | | little | m6.6-eevdf-complete-no-delay-zero | load | 34.51% | | little | m6.6-eevdf-complete-no-delay-zero | util | 72.53% | | little | m6.6-eevdf-complete-pelt-fix | load | 29.96% | | little | m6.6-eevdf-complete-pelt-fix | util | 59.82% | | mid | m6.6-stock | load | 0.0% | | mid | m6.6-stock | util | 0.0% | | mid | m6.6-eevdf-complete | load | 29.37% | | mid | m6.6-eevdf-complete | util | 75.22% | | mid | m6.6-eevdf-complete-no-delay-dequeue | load | 36.4% | | mid | m6.6-eevdf-complete-no-delay-dequeue | util | 80.28% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 30.35% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 90.2% | | mid | m6.6-eevdf-complete-no-delay-zero | load | 37.83% | | mid | m6.6-eevdf-complete-no-delay-zero | util | 93.79% | | mid | m6.6-eevdf-complete-pelt-fix | load | 33.57% | | mid | m6.6-eevdf-complete-pelt-fix | util | 67.83% | | big | m6.6-stock | load | 0.0% | | big | m6.6-stock | util | 0.0% | | big | m6.6-eevdf-complete | load | 97.39% | | big | m6.6-eevdf-complete | util | 12.63% | | big | m6.6-eevdf-complete-no-delay-dequeue | load | 139.69% | | big | m6.6-eevdf-complete-no-delay-dequeue | util | 22.58% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 125.36% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 23.15% | | big | m6.6-eevdf-complete-no-delay-zero | load | 128.56% | | big | m6.6-eevdf-complete-no-delay-zero | util | 25.03% | | big | m6.6-eevdf-complete-pelt-fix | load | 130.73% | | big | m6.6-eevdf-complete-pelt-fix | util | 17.52% | +---------+------------------------------------------------------+----------+-----------+
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 118 lines · Luis Machado
Hi Peter,
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.Quoted 8 lines
Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
I gave this a try on the Pixel 6, and I noticed some improvement (see below), but not enough to bring it back to the original levels.
(1) m6.6-stock - Basic EEVDF with wakeup preemption fix (baseline)
(2) m6.6-eevdf-complete: m6.6-stock plus this series.
(3) m6.6-eevdf-complete-no-delay-dequeue: (2) +NO_DELAY_DEQUEUE
(4) m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero: (2) +NO_DELAY_DEQUEUE+NO_DELAY_ZERO
(5) m6.6-eevdf-complete-no-delay-zero: (2) +NO_DELAY_ZERO
(6) m6.6-eevdf-complete-pelt-fix: (2) + the proposedload_avgupdate patch.I included (3), (4) and (5) to exercise the impact of disabling the individual scheduler features.
Energy use.
+------------+------------------------------------------------------+-----------+ | cluster | tag | perc_diff | +------------+------------------------------------------------------+-----------+ | CPU | m6.6-stock | 0.0% | | CPU-Big | m6.6-stock | 0.0% | | CPU-Little | m6.6-stock | 0.0% | | CPU-Mid | m6.6-stock | 0.0% | | GPU | m6.6-stock | 0.0% | | Total | m6.6-stock | 0.0% | | CPU | m6.6-eevdf-complete | 114.51% | | CPU-Big | m6.6-eevdf-complete | 90.75% | | CPU-Little | m6.6-eevdf-complete | 98.74% | | CPU-Mid | m6.6-eevdf-complete | 213.9% | | GPU | m6.6-eevdf-complete | -7.04% | | Total | m6.6-eevdf-complete | 100.92% | | CPU | m6.6-eevdf-complete-no-delay-dequeue | 117.77% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue | 113.79% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue | 97.47% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue | 189.0% | | GPU | m6.6-eevdf-complete-no-delay-dequeue | -6.74% | | Total | m6.6-eevdf-complete-no-delay-dequeue | 103.84% | | CPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 120.45% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 113.65% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 99.04% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 201.14% | | GPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | -5.37% | | Total | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 106.38% | | CPU | m6.6-eevdf-complete-no-delay-zero | 119.05% | | CPU-Big | m6.6-eevdf-complete-no-delay-zero | 107.55% | | CPU-Little | m6.6-eevdf-complete-no-delay-zero | 98.66% | | CPU-Mid | m6.6-eevdf-complete-no-delay-zero | 206.58% | | GPU | m6.6-eevdf-complete-no-delay-zero | -5.25% | | Total | m6.6-eevdf-complete-no-delay-zero | 105.14% | | CPU | m6.6-eevdf-complete-pelt-fix | 105.56% | | CPU-Big | m6.6-eevdf-complete-pelt-fix | 100.45% | | CPU-Little | m6.6-eevdf-complete-pelt-fix | 94.4% | | CPU-Mid | m6.6-eevdf-complete-pelt-fix | 150.94% | | GPU | m6.6-eevdf-complete-pelt-fix | -3.96% | | Total | m6.6-eevdf-complete-pelt-fix | 93.31% | +------------+------------------------------------------------------+-----------+Utilization and load levels.
+---------+------------------------------------------------------+----------+-----------+ | cluster | tag | variable | perc_diff | +---------+------------------------------------------------------+----------+-----------+ | little | m6.6-stock | load | 0.0% | | little | m6.6-stock | util | 0.0% | | little | m6.6-eevdf-complete | load | 29.56% | | little | m6.6-eevdf-complete | util | 55.4% | | little | m6.6-eevdf-complete-no-delay-dequeue | load | 42.89% | | little | m6.6-eevdf-complete-no-delay-dequeue | util | 69.47% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 51.05% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 76.55% | | little | m6.6-eevdf-complete-no-delay-zero | load | 34.51% | | little | m6.6-eevdf-complete-no-delay-zero | util | 72.53% | | little | m6.6-eevdf-complete-pelt-fix | load | 29.96% | | little | m6.6-eevdf-complete-pelt-fix | util | 59.82% | | mid | m6.6-stock | load | 0.0% | | mid | m6.6-stock | util | 0.0% | | mid | m6.6-eevdf-complete | load | 29.37% | | mid | m6.6-eevdf-complete | util | 75.22% | | mid | m6.6-eevdf-complete-no-delay-dequeue | load | 36.4% | | mid | m6.6-eevdf-complete-no-delay-dequeue | util | 80.28% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 30.35% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 90.2% | | mid | m6.6-eevdf-complete-no-delay-zero | load | 37.83% | | mid | m6.6-eevdf-complete-no-delay-zero | util | 93.79% | | mid | m6.6-eevdf-complete-pelt-fix | load | 33.57% | | mid | m6.6-eevdf-complete-pelt-fix | util | 67.83% | | big | m6.6-stock | load | 0.0% | | big | m6.6-stock | util | 0.0% | | big | m6.6-eevdf-complete | load | 97.39% | | big | m6.6-eevdf-complete | util | 12.63% | | big | m6.6-eevdf-complete-no-delay-dequeue | load | 139.69% |
Quoted 8 lines
| big | m6.6-eevdf-complete-no-delay-dequeue | util | 22.58% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 125.36% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 23.15% | | big | m6.6-eevdf-complete-no-delay-zero | load | 128.56% | | big | m6.6-eevdf-complete-no-delay-zero | util | 25.03% | | big | m6.6-eevdf-complete-pelt-fix | load | 130.73% | | big | m6.6-eevdf-complete-pelt-fix | util | 17.52% | +---------+------------------------------------------------------+----------+-----------+
Going through the code, my understanding is that the util_est functions seem to be getting called correctly, and in the right order. That is, we first util_est_enqueue, then util_est_dequeue and finally util_est_update. So the stats *should* be correct.
On dequeuing (dequeue_task_fair), we immediately call util_est_dequeue, even for the case of a DEQUEUE_DELAYED task, since we're no longer going to run the dequeue_delayed task for now, even though it is still in the rq.
We delay the util_est_update of dequeue_delayed tasks until a later time in dequeue_entities.
Eventually the dequeue_delayed task will have its lag zeroed when it becomes eligible again, (requeue_delayed_entity) while still being in the rq. It will then get dequeued/enqueued (requeued), and marked as a non-dequeue-delayed task.
Next time we attempt to enqueue such a task (enqueue_task_fair), it will skip the ENQUEUE_DELAYED block and call util_est_enqueue.
Still, something seems to be signalling that util/load is high, and causing migration to the big cores.
Maybe we're not decaying the util/load properly at some point, and inflated numbers start to happen.
I'll continue investigating.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 142 lines · Luis Machado
Quoted 118 lines · Luis Machado
Hi Peter,
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.The PELT runnable values will be inflated because of delayed dequeue.
cpu_util()uses those in the @boost case, and as such this can indeed affect things.Quoted 8 lines
Quoted 5 lines
This can also slightly affect the cgroup case, but since the delay goes away as contention goes away, and the cgroup case must already assume worst case overlap, this seems limited.
/me goes ponder things moar.
First order approximation of a fix would be something like the totally untested below I suppose...
I gave this a try on the Pixel 6, and I noticed some improvement (see below), but not enough to bring it back to the original levels.
(1) m6.6-stock - Basic EEVDF with wakeup preemption fix (baseline)
(2) m6.6-eevdf-complete: m6.6-stock plus this series.
(3) m6.6-eevdf-complete-no-delay-dequeue: (2) +NO_DELAY_DEQUEUE
(4) m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero: (2) +NO_DELAY_DEQUEUE+NO_DELAY_ZERO
(5) m6.6-eevdf-complete-no-delay-zero: (2) +NO_DELAY_ZERO
(6) m6.6-eevdf-complete-pelt-fix: (2) + the proposedload_avgupdate patch.I included (3), (4) and (5) to exercise the impact of disabling the individual scheduler features.
Energy use.
+------------+------------------------------------------------------+-----------+ | cluster | tag | perc_diff | +------------+------------------------------------------------------+-----------+ | CPU | m6.6-stock | 0.0% | | CPU-Big | m6.6-stock | 0.0% | | CPU-Little | m6.6-stock | 0.0% | | CPU-Mid | m6.6-stock | 0.0% | | GPU | m6.6-stock | 0.0% | | Total | m6.6-stock | 0.0% | | CPU | m6.6-eevdf-complete | 114.51% | | CPU-Big | m6.6-eevdf-complete | 90.75% | | CPU-Little | m6.6-eevdf-complete | 98.74% | | CPU-Mid | m6.6-eevdf-complete | 213.9% | | GPU | m6.6-eevdf-complete | -7.04% | | Total | m6.6-eevdf-complete | 100.92% | | CPU | m6.6-eevdf-complete-no-delay-dequeue | 117.77% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue | 113.79% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue | 97.47% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue | 189.0% | | GPU | m6.6-eevdf-complete-no-delay-dequeue | -6.74% | | Total | m6.6-eevdf-complete-no-delay-dequeue | 103.84% | | CPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 120.45% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 113.65% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 99.04% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 201.14% | | GPU | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | -5.37% | | Total | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | 106.38% | | CPU | m6.6-eevdf-complete-no-delay-zero | 119.05% | | CPU-Big | m6.6-eevdf-complete-no-delay-zero | 107.55% | | CPU-Little | m6.6-eevdf-complete-no-delay-zero | 98.66% | | CPU-Mid | m6.6-eevdf-complete-no-delay-zero | 206.58% | | GPU | m6.6-eevdf-complete-no-delay-zero | -5.25% | | Total | m6.6-eevdf-complete-no-delay-zero | 105.14% | | CPU | m6.6-eevdf-complete-pelt-fix | 105.56% | | CPU-Big | m6.6-eevdf-complete-pelt-fix | 100.45% | | CPU-Little | m6.6-eevdf-complete-pelt-fix | 94.4% | | CPU-Mid | m6.6-eevdf-complete-pelt-fix | 150.94% | | GPU | m6.6-eevdf-complete-pelt-fix | -3.96% | | Total | m6.6-eevdf-complete-pelt-fix | 93.31% | +------------+------------------------------------------------------+-----------+Utilization and load levels.
+---------+------------------------------------------------------+----------+-----------+ | cluster | tag | variable | perc_diff | +---------+------------------------------------------------------+----------+-----------+ | little | m6.6-stock | load | 0.0% | | little | m6.6-stock | util | 0.0% | | little | m6.6-eevdf-complete | load | 29.56% | | little | m6.6-eevdf-complete | util | 55.4% | | little | m6.6-eevdf-complete-no-delay-dequeue | load | 42.89% | | little | m6.6-eevdf-complete-no-delay-dequeue | util | 69.47% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 51.05% | | little | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 76.55% | | little | m6.6-eevdf-complete-no-delay-zero | load | 34.51% | | little | m6.6-eevdf-complete-no-delay-zero | util | 72.53% | | little | m6.6-eevdf-complete-pelt-fix | load | 29.96% | | little | m6.6-eevdf-complete-pelt-fix | util | 59.82% | | mid | m6.6-stock | load | 0.0% | | mid | m6.6-stock | util | 0.0% | | mid | m6.6-eevdf-complete | load | 29.37% | | mid | m6.6-eevdf-complete | util | 75.22% | | mid | m6.6-eevdf-complete-no-delay-dequeue | load | 36.4% | | mid | m6.6-eevdf-complete-no-delay-dequeue | util | 80.28% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 30.35% | | mid | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 90.2% | | mid | m6.6-eevdf-complete-no-delay-zero | load | 37.83% | | mid | m6.6-eevdf-complete-no-delay-zero | util | 93.79% | | mid | m6.6-eevdf-complete-pelt-fix | load | 33.57% | | mid | m6.6-eevdf-complete-pelt-fix | util | 67.83% | | big | m6.6-stock | load | 0.0% | | big | m6.6-stock | util | 0.0% | | big | m6.6-eevdf-complete | load | 97.39% | | big | m6.6-eevdf-complete | util | 12.63% | | big | m6.6-eevdf-complete-no-delay-dequeue | load | 139.69% |Quoted 8 lines
| big | m6.6-eevdf-complete-no-delay-dequeue | util | 22.58% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | load | 125.36% | | big | m6.6-eevdf-complete-no-delay-dequeue-no-delay-zero | util | 23.15% | | big | m6.6-eevdf-complete-no-delay-zero | load | 128.56% | | big | m6.6-eevdf-complete-no-delay-zero | util | 25.03% | | big | m6.6-eevdf-complete-pelt-fix | load | 130.73% | | big | m6.6-eevdf-complete-pelt-fix | util | 17.52% | +---------+------------------------------------------------------+----------+-----------+Going through the code, my understanding is that the
util_estfunctions seem to be getting called correctly, and in the right order. That is, we firstutil_est_enqueue, thenutil_est_dequeueand finallyutil_est_update. So the stats *should* be correct.On dequeuing (
dequeue_task_fair), we immediately callutil_est_dequeue, even for the case of aDEQUEUE_DELAYEDtask, since we're no longer going to run thedequeue_delayedtask for now, even though it is still in the rq.We delay the
util_est_updateofdequeue_delayedtasks until a later time indequeue_entities.Eventually the
dequeue_delayedtask will have its lag zeroed when it becomes eligible again, (requeue_delayed_entity) while still being in the rq. It will then get dequeued/enqueued (requeued), and marked as a non-dequeue-delayed task.
Quoted 9 lines
Next time we attempt to enqueue such a task (
enqueue_task_fair), it will skip theENQUEUE_DELAYEDblock and callutil_est_enqueue.Still, something seems to be signalling that util/load is high, and causing migration to the big cores.
Maybe we're not decaying the util/load properly at some point, and inflated numbers start to happen.
I'll continue investigating.
Just a quick update on this. While investigating this behavior, I spotted very high loadavg values on an idle system. For instance:
load average: 4733.84, 4721.24, 4680.33
I wonder if someone else also spotted this.
These values keep increasing slowly but steadily. When the system is under load they increase a bit more rapidly. Makes me wonder if we're missing decrementing nr_uninterruptible in some path, since that is what seems to be causing the loadavg to be off.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 6 lines · Luis Machado
Just a quick update on this. While investigating this behavior, I spotted very high loadavg values on an idle system. For instance:
load average: 4733.84, 4721.24, 4680.33
I wonder if someone else also spotted this.
Hadn't spotted it, but now that you mention it, I can definitely see it.
Let me go prod with something sharp. Thanks!
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Luis Machado
Just a quick update on this. While investigating this behavior, I spotted very high loadavg values on an idle system. For instance:
load average: 4733.84, 4721.24, 4680.33
I wonder if someone else also spotted this.
Hadn't spotted it, but now that you mention it, I can definitely see it.
Let me go prod with something sharp. Thanks!
What's the point of making notes if you then don't read them... *sigh*.
Does this help?
--- kernel/sched/core.c | 23 ++++++++++++----------- kernel/sched/fair.c | 4 ++-- kernel/sched/sched.h | 8 ++++++++ 3 files changed, 22 insertions(+), 13 deletions(-)
kernel/sched/core.c +12 -11
@@ -2154,14 +2154,18 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)void deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -bool sleep = flags &DEQUEUE_SLEEP ; +SCHED_WARN_ON (flags &DEQUEUE_SLEEP ); -if (dequeue_task (rq, p, flags)) { -WRITE_ONCE (p->on_rq, sleep ?0 :TASK_ON_RQ_MIGRATING ); -ASSERT_EXCLUSIVE_WRITER (p->on_rq); - }else { -SCHED_WARN_ON (!sleep);/* only sleep can fail */ - } +dequeue_task (rq, p, flags); + +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +} + +static void block_task (struct rq *rq,struct task_struct *p,int flags) +{ +if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +__block_task (rq, p); }static inline int __normal_prio (int policy,int rt_prio,int nice)
@@ -6693,9 +6697,6 @@ static void __sched notrace __schedule(unsigned int sched_mode) !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); -if (prev->sched_contributes_to_load) - rq->nr_uninterruptible++; -/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...)
@@ -6707,7 +6708,7 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * After this, schedule() must not care about p->state any more. */ -deactivate_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK ); +block_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK );if (prev->in_iowait) {atomic_inc (&rq->nr_iowait);
kernel/sched/fair.c +2 -2
@@ -7032,8 +7032,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); -/* Fix-up what deactivate_task() skipped. */ -WRITE_ONCE (p->on_rq,0 ); +/* Fix-up what block_task() skipped. */ +__block_task (rq, p); } }
kernel/sched/sched.h +8 -0
@@ -2560,6 +2560,14 @@ static inline void sub_nr_running(struct rq *rq, unsigned count)sched_update_tick_dependency (rq); } +static inline void __block_task (struct rq *rq,struct task_struct *p) +{ +WRITE_ONCE (p->on_rq,0 ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; +} +extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Luis Machado
Just a quick update on this. While investigating this behavior, I spotted very high loadavg values on an idle system. For instance:
load average: 4733.84, 4721.24, 4680.33
I wonder if someone else also spotted this.
Hadn't spotted it, but now that you mention it, I can definitely see it.
(ditto.. wart not enormous enough for awesome powers of observation:)
Quoted 6 lines
Quoted 2 lines
Let me go prod with something sharp. Thanks!
What's the point of making notes if you then don't read them... *sigh*.
Does this help?
That fixed it up here.
Quoted 89 lines
---
kernel/sched/core.c | 23 ++++++++++++----------- kernel/sched/fair.c | 4 ++--kernel/sched/sched.h| 8 ++++++++3 files changed, 22 insertions(+), 13 deletions(-)kernel/sched/core.c +0
-0@@ -2154,14 +2154,18 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)
voiddeactivate_task(struct rq *rq, struct task_struct *p, int flags){
- bool sleep = flags &DEQUEUE_SLEEP;
+ SCHED_WARN_ON(flags &DEQUEUE_SLEEP);
- if (dequeue_task(rq, p, flags)) {
- WRITE_ONCE(p->on_rq, sleep ? 0 :TASK_ON_RQ_MIGRATING); - ASSERT_EXCLUSIVE_WRITER(p->on_rq);
- } else {
- SCHED_WARN_ON(!sleep); /* only sleep can fail */ - }
+ dequeue_task(rq, p, flags);kernel/sched/core.c +9
-0+ + WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); +} + +static void block_task (struct rq *rq,struct task_struct *p,int flags) +{ + if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags)) + __block_task(rq, p);}
static inline int__normal_prio(int policy, int rt_prio, int nice)kernel/sched/core.c +0
-0@@ -6693,9 +6697,6 @@ static void __sched notrace __schedule(unsigned int sched_mode)!(prev_state &
TASK_NOLOAD) && !(prev_state &TASK_FROZEN);
- if (prev->sched_contributes_to_load)
- rq->nr_uninterruptible++;
-
/*
* __schedule() ttwu()
*prev_state= prev->state; if (p->on_rq&& ...)kernel/sched/core.c +1
-1@@ -6707,7 +6708,7 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * After this, schedule() must not care about p->state any more. */ - deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); + block_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK);
if (prev->in_iowait) {atomic_inc(&rq->nr_iowait);kernel/sched/fair.c +0
-0@@ -7032,8 +7032,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) util_est_update(&rq->cfs, p, task_sleep); hrtick_update(rq);
- /* Fix-up whatdeactivate_task()skipped. */ - WRITE_ONCE(p->on_rq, 0);
+ /* Fix-up whatblock_task()skipped. */ + __block_task(rq, p);
}
}kernel/sched/sched.h +0
-0@@ -2560,6 +2560,14 @@ static inline void sub_nr_running(struct rq *rq, unsigned count) sched_update_tick_dependency(rq); }kernel/sched/sched.h +8
-0+static inline void __block_task (struct rq *rq,struct task_struct *p) +{ + WRITE_ONCE(p->on_rq, 0); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); + if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; +} +extern void
activate_task(struct rq *rq, struct task_struct *p, int flags); extern voiddeactivate_task(struct rq *rq, struct task_struct *p, int flags);
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Luis Machado
Just a quick update on this. While investigating this behavior, I spotted very high loadavg values on an idle system. For instance:
load average: 4733.84, 4721.24, 4680.33
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I wonder if someone else also spotted this.
Hadn't spotted it, but now that you mention it, I can definitely see it.
Let me go prod with something sharp. Thanks!
What's the point of making notes if you then don't read them... *sigh*.
Makes it look like you did read them? :-)
Quoted 3 lines
Does this help?
It does address the load_avg issues as Mike G pointed out. Thanks for the quick patch!
Unfortunately, for me, it didn't help with the energy regression. So my investigation continues.
Something still seems to be driving things out of the smaller cores and into the bigger cores. And the load_avg accounting proved to be a red herring.
Quoted 89 lines
--- kernel/sched/core.c | 23 ++++++++++++----------- kernel/sched/fair.c | 4 ++-- kernel/sched/sched.h | 8 ++++++++ 3 files changed, 22 insertions(+), 13 deletions(-)kernel/sched/core.c +12
-11@@ -2154,14 +2154,18 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)void deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -bool sleep = flags &DEQUEUE_SLEEP ; +SCHED_WARN_ON (flags &DEQUEUE_SLEEP ); -if (dequeue_task (rq, p, flags)) { -WRITE_ONCE (p->on_rq, sleep ?0 :TASK_ON_RQ_MIGRATING ); -ASSERT_EXCLUSIVE_WRITER (p->on_rq); - }else { -SCHED_WARN_ON (!sleep);/* only sleep can fail */ - } +dequeue_task (rq, p, flags); + +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +} + +static void block_task (struct rq *rq,struct task_struct *p,int flags) +{ +if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +__block_task (rq, p); }static inline int __normal_prio (int policy,int rt_prio,int nice)@@ -6693,9 +6697,6 @@ static void __sched notrace __schedule(unsigned int sched_mode) !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); -if (prev->sched_contributes_to_load) - rq->nr_uninterruptible++; -/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...) @@ -6707,7 +6708,7 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * After this, schedule() must not care about p->state any more. */ -deactivate_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK ); +block_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK );if (prev->in_iowait) {atomic_inc (&rq->nr_iowait);kernel/sched/fair.c +2
-2@@ -7032,8 +7032,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); -/* Fix-up what deactivate_task() skipped. */ -WRITE_ONCE (p->on_rq,0 ); +/* Fix-up what block_task() skipped. */ +__block_task (rq, p); } }kernel/sched/sched.h +8
-0@@ -2560,6 +2560,14 @@ static inline void sub_nr_running(struct rq *rq, unsigned count)sched_update_tick_dependency (rq); } +static inline void __block_task (struct rq *rq,struct task_struct *p) +{ +WRITE_ONCE (p->on_rq,0 ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; +} +extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Luis MachadoQuoted 14 lines · Luis Machado
Going through the code, my understanding is that the
util_estfunctions seem to be getting called correctly, and in the right order. That is, we firstutil_est_enqueue, thenutil_est_dequeueand finallyutil_est_update. So the stats *should* be correct.On dequeuing (
dequeue_task_fair), we immediately callutil_est_dequeue, even for the case of aDEQUEUE_DELAYEDtask, since we're no longer going to run thedequeue_delayedtask for now, even though it is still in the rq.We delay the
util_est_updateofdequeue_delayedtasks until a later time indequeue_entities.Eventually the
dequeue_delayedtask will have its lag zeroed when it becomes eligible again, (requeue_delayed_entity) while still being in the rq. It will then get dequeued/enqueued (requeued), and marked as a non-dequeue-delayed task.
Quoted 8 lines
Next time we attempt to enqueue such a task (
enqueue_task_fair), it will skip theENQUEUE_DELAYEDblock and callutil_est_enqueue.Still, something seems to be signalling that util/load is high, and causing migration to the big cores.
Maybe we're not decaying the util/load properly at some point, and inflated numbers start to happen.
I'll continue investigating.
So I re-read all this util_est stuff again this evening and I am confused :-) Please bear with me.
So the old code does:
dequeue_task_fair() util_est_dequeue(); // actual dequeue -- updates pelt util_est_update();
While the new code does:
dequeue_task_fair() util_est_dequeue(); if (!dequeue()) return; util_est_update(); delayed_dequeue: util_est_update();
Specifically, we only call util_est_update() if/when we do the actual dequeue -- potentially at a later point in time. Because, as I argued in the comments, ttwu()'s ENQUEUE_DELAYED will not actually enqueue the task (since it is still enqueued) and therefore the update would be spurious.
However!!, if we do dequeue, then we'll end up updating the EWMA with a potentially different task_util(p).
And this is where the confusion comes... this extra time on the runqueue would not be running and thus decreate util_avg, as such task_util_est() should be lower than before and tasks should tend towards smaller cores, rather than larger cores as you seem to be seeing.
[ there is the 'dequeued + UTIL_EST_MARGIN < task_runnable()' escape
clause, which we might be hitting... dunno ]
In any case, if you haven't tried it already, could I ask you to test the below (once you're back in the office)?
Also, this doesn't really explain why things go sideways once you enable DELAY_DEQUEUE and then don't re-align if you disable it again. I mean, it should eventually age out the dodgy EWMA contributions and start working as expected.
kernel/sched/fair.c +3 -6
@@ -7008,7 +7008,6 @@ static int dequeue_entities(struct rq *rSCHED_WARN_ON (p->on_rq !=1 );/* Fix-up what dequeue_task_fair() skipped */ -util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq);/* Fix-up what block_task() skipped. */
@@ -7028,13 +7027,11 @@ static bool dequeue_task_fair(struct rq {util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (dequeue_entities (rq, &p->se, flags) <0 ) { +util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );return false; + } -/* -* It doesn't make sense to update util_est for the delayed dequeue -* case where ttwu will make it appear the sleep never happened. -*/ util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);return true;
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Luis Machado
Going through the code, my understanding is that the
util_estfunctions seem to be getting called correctly, and in the right order. That is, we firstutil_est_enqueue, thenutil_est_dequeueand finallyutil_est_update. So the stats *should* be correct.On dequeuing (
dequeue_task_fair), we immediately callutil_est_dequeue, even for the case of aDEQUEUE_DELAYEDtask, since we're no longer going to run thedequeue_delayedtask for now, even though it is still in the rq.We delay the
util_est_updateofdequeue_delayedtasks until a later time indequeue_entities.Eventually the
dequeue_delayedtask will have its lag zeroed when it becomes eligible again, (requeue_delayed_entity) while still being in the rq. It will then get dequeued/enqueued (requeued), and marked as a non-dequeue-delayed task.
Quoted 8 lines
Quoted 8 lines
Next time we attempt to enqueue such a task (
enqueue_task_fair), it will skip theENQUEUE_DELAYEDblock and callutil_est_enqueue.Still, something seems to be signalling that util/load is high, and causing migration to the big cores.
Maybe we're not decaying the util/load properly at some point, and inflated numbers start to happen.
I'll continue investigating.
Apologies for not getting this earlier, but I wanted to collect as much data as possible so we can hopefully see what's going wrong with this particular patch in the series.
(picking up from the previous questions on find_energy_efficient_cpu and overutilized)
For the rate of calls to find_energy_efficient_cpu, I checked again and I didn't see any significant differences there. With delayed dequeue or without delayed dequeue, I see almost the same number of calls, with the expected small variation between runs.
As for the overutilized part, there is a bit of a change within find_energy_efficient_cpu. When I first enable delayed dequeue via the debugfs (I boot up the system with it disabled for a clean state), I start to see a lot more cases of us bailing out of find_energy_efficient_cpu due to the overutilized condition.
Strangely though, things seem to stabilize after a while, and the rate of overutilized conditions when within find_energy_efficient_cpu goes down to normal levels.
Here's some numbers from a trace session:
Delayed dequeue off (clean state, 2 iterations):
------------------------------------------------
Calls to find_energy_efficient_cpu: 211087 Overutilized when in find_energy_efficient_cpu: 36224
Calls to find_energy_efficient_cpu: 205804 Overutilized when in find_energy_efficient_cpu: 35274
Delayed dequeue on (3 iterations):
------------------------------------------------
Calls to find_energy_efficient_cpu: 204079 Overutilized when in find_energy_efficient_cpu: 86966
Calls to find_energy_efficient_cpu: 207351 Overutilized when in find_energy_efficient_cpu: 111233
Calls to find_energy_efficient_cpu: 208556 Overutilized when in find_energy_efficient_cpu: 77986
Delayed dequeue off again (3 iterations):
------------------------------------------------
Calls to find_energy_efficient_cpu: 210396 Overutilized when in find_energy_efficient_cpu: 34108
Calls to find_energy_efficient_cpu: 210978 Overutilized when in find_energy_efficient_cpu: 33122
Calls to find_energy_efficient_cpu: 210181 Overutilized when in find_energy_efficient_cpu: 33060
Delayed dequeue on again (3 iterations):
------------------------------------------------
Calls to find_energy_efficient_cpu: 202932 Overutilized when in find_energy_efficient_cpu: 35060
Calls to find_energy_efficient_cpu: 206098 Overutilized when in find_energy_efficient_cpu: 57373
Calls to find_energy_efficient_cpu: 199444 Overutilized when in find_energy_efficient_cpu: 33959
On the matter of switched_from_fair() being called with a task that is sched_delayed == 1, I can confirm I see some occurrences of that. They are not that numerous, but do happen from time to time on the trace data. I haven't investigated things in that direction yet though.
But we do make those tasks sched_delayed = 0 in there, so things should go back to normal eventually, unless we fail to dequeue the sched_delayed task properly for this case.
Quoted 14 lines
So I re-read all this
util_eststuff again this evening and I am confused :-) Please bear with me.So the old code does:
dequeue_task_fair() util_est_dequeue(); // actual dequeue -- updates pelt util_est_update();While the new code does:
Quoted 10 lines
dequeue_task_fair() util_est_dequeue(); if (!dequeue()) return; util_est_update(); delayed_dequeue: util_est_update();
The above is my understanding as well.
Quoted 6 lines
Specifically, we only call
util_est_update()if/when we do the actual dequeue -- potentially at a later point in time. Because, as I argued in the comments, ttwu()'sENQUEUE_DELAYEDwill not actually enqueue the task (since it is still enqueued) and therefore the update would be spurious.
Quoted 7 lines
However!!, if we do dequeue, then we'll end up updating the EWMA with a potentially different
task_util(p).And this is where the confusion comes... this extra time on the runqueue would not be running and thus decreate
util_avg, as suchtask_util_est()should be lower than before and tasks should tend towards smaller cores, rather than larger cores as you seem to be seeing.
Right. I also expected the util_avg to decrease with more time on the runqueue. And that might be happening...
Quoted 29 lines
[ there is the 'dequeued +
UTIL_EST_MARGIN<task_runnable()' escape
clause, which we might be hitting... dunno ]In any case, if you haven't tried it already, could I ask you to test the below (once you're back in the office)?
Also, this doesn't really explain why things go sideways once you enable
DELAY_DEQUEUEand then don't re-align if you disable it again. I mean, it should eventually age out the dodgy EWMA contributions and start working as expected.kernel/sched/fair.c +2
-2@@ -7008,7 +7008,6 @@ static int dequeue_entities(struct rq *rSCHED_WARN_ON (p->on_rq !=1 );/* Fix-up what dequeue_task_fair() skipped */ -util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq);/* Fix-up what block_task() skipped. */ @@ -7028,13 +7027,11 @@ static bool dequeue_task_fair(struct rq {util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (dequeue_entities (rq, &p->se, flags) <0 ) { +util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );
kernel/sched/fair.c static bool dequeue_task_fair(struct rq · line 7037
Quoted 10 lines
kernel/sched/fair.c +1
-4return false; + } -/* -* It doesn't make sense to update util_est for the delayed dequeue -* case where ttwu will make it appear the sleep never happened. -*/ util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);return true;
... as the above patch unfortunately doesn't improve things. The util_est_update might be a red herring here.
Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp behavior with the eevdf complete series applied.
Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing of tasks, and the uclamp values are organized in buckets.
Just for fun I added a few trace_printk's in uclamp_eff_value, uclamp_rq_inc_id and uclamp_rq_dec_id.
Booting up the system with delayed_dequeue disabled and running the benchmark, I see the uclamp bucket management pretty stable. Tasks get added to the uclamp buckets but then get removed. At the end of the benchmark, the uclamp buckets are (almost always) clean of tasks.
Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50 tasks still. If I do another run, I can see 80, 100 tasks still.
I suspect refcounting might be going wrong somewhere due to delayed dequeue tasks, but that's more of a guess right now. Hopefully that is useful information. I'll resume investigation tomorrow.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 14 lines · Luis Machado
Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp behavior with the eevdf complete series applied.
Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing of tasks, and the uclamp values are organized in buckets.
Just for fun I added a few trace_printk's in
uclamp_eff_value,uclamp_rq_inc_idanduclamp_rq_dec_id.Booting up the system with
delayed_dequeuedisabled and running the benchmark, I see the uclamp bucket management pretty stable. Tasks get added to the uclamp buckets but then get removed. At the end of the benchmark, the uclamp buckets are (almost always) clean of tasks.
Quoted 7 lines
Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50 tasks still. If I do another run, I can see 80, 100 tasks still.
I suspect refcounting might be going wrong somewhere due to delayed dequeue tasks, but that's more of a guess right now. Hopefully that is useful information. I'll resume investigation tomorrow.
Thank you both!!
Does the below help?
Note how dequeue_task() does uclamp_rq_dec() unconditionally, which is then not balanced in the case below.
kernel/sched/core.c +1 -0
@@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str/* mustn't run a delayed task */ SCHED_WARN_ON (task_on_cpu (rq, p));enqueue_task (rq, p,ENQUEUE_DELAYED ); +uclamp_rq_inc (rq, p); }if (!task_on_cpu (rq, p)) {/*
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Hi Peter,
Quoted 34 lines · Peter Zijlstra
Quoted 14 lines · Luis Machado
Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp behavior with the eevdf complete series applied.
Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing of tasks, and the uclamp values are organized in buckets.
Just for fun I added a few trace_printk's in
uclamp_eff_value,uclamp_rq_inc_idanduclamp_rq_dec_id.Booting up the system with
delayed_dequeuedisabled and running the benchmark, I see the uclamp bucket management pretty stable. Tasks get added to the uclamp buckets but then get removed. At the end of the benchmark, the uclamp buckets are (almost always) clean of tasks.Quoted 7 lines
Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50 tasks still. If I do another run, I can see 80, 100 tasks still.
I suspect refcounting might be going wrong somewhere due to delayed dequeue tasks, but that's more of a guess right now. Hopefully that is useful information. I'll resume investigation tomorrow.
Thank you both!!
Does the below help?
Note how
dequeue_task()doesuclamp_rq_dec()unconditionally, which is then not balanced in the case below.
kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3670
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str/* mustn't run a delayed task */ SCHED_WARN_ON (task_on_cpu (rq, p));enqueue_task (rq, p,ENQUEUE_DELAYED ); +uclamp_rq_inc (rq, p); }if (!task_on_cpu (rq, p)) {/*
Further communication with Luis makes me investigate more on uclamp bucket leaks.
I added a separate variable uclamp_on_rq in task_struct and WARN when the state didn't match. The thing is, without your above fix, the first WARN happens on uclamp_rq_inc(), not dec(), which means a task is already added to rq uclamp and is somehow added again by mistake.
Because this is a double inc(), your fix won't fix this problem. I added the trace here when the double inc() happened, in case you can see something.
Juno-r2-Debian login: [ 176.010780] ------------[ cut here ]------------
[1083/15764] [ 176.015414] WARNING: CPU: 5 PID: 0 at kernel/sched/core.c:1594 uclamp_rq_inc+0x188/0x1a0
[ 176.023539] Modules linked in:
[ 176.026602] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 6.10.0-rc1-ga946a2b48119-dirty #1034
[ 176.034882] Hardware name: ARM Juno development board (r2) (DT)
[ 176.040815] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 176.047796] pc : uclamp_rq_inc+0x188/0x1a0
[ 176.051903] lr : enqueue_task+0x48/0x70
[ 176.055752] sp : ffffffc081fb3de0
[ 176.059073] x29: ffffffc081fb3de0 x28: 0000000000000000 x27: ffffff8800342300
[ 176.066233] x26: ffffff88084311b8 x25: ffffffc081ed2000 x24: ffffffc081ca7008
[ 176.073388] x23: 0000000000000000 x22: 00000000000000c0 x21: ffffff897efbbf40
[ 176.080543] x20: ffffff8808431180 x19: 0000000000000049 x18: 0000000000000408
[ 176.087700] x17: ffffffc8fd301000 x16: ffffffc081fb0000 x15: 000000000000afc8
[ 176.094857] x14: 0000000000000001 x13: 0000000000000004 x12: 000000020c8392c2
[ 176.102017] x11: 0000000000000800 x10: 0000000000000800 x9 : 0000000000000000
[ 176.109177] x8 : 0000000000000001 x7 : 0000000000000021 x6 : ffffffffffffff4c
[ 176.116337] x5 : 0000000000007e90 x4 : 00000000000fc33c x3 : 00000000000139b4
[ 176.123487] x2 : 0000000000000049 x1 : ffffff8808431180 x0 : ffffff897efbbf40
[ 176.130639] Call trace:
[ 176.133089] uclamp_rq_inc+0x188/0x1a0
[ 176.136849] activate_task+0x48/0x60
[ 176.140433] ttwu_do_activate+0x80/0x22c
[ 176.144364] sched_ttwu_pending+0x9c/0x120
[ 176.148470] __flush_smp_call_function_queue+0x3d0/0x568
[ 176.153792] generic_smp_call_function_single_interrupt+0x10/0x18
[ 176.159902] do_handle_IPI+0xa0/0x230
[ 176.163577] ipi_handler+0x18/0x24
[ 176.166987] handle_percpu_devid_irq+0x80/0x204
[ 176.171532] generic_handle_domain_irq+0x28/0x40
[ 176.176161] gic_handle_irq+0x58/0xd0
[ 176.179833] call_on_irq_stack+0x24/0x34
[ 176.183764] el1_interrupt+0x70/0xbc
[ 176.187348] el1h_64_irq_handler+0x14/0x1c
[ 176.191449] el1h_64_irq+0x64/0x68
[ 176.194857] arch_local_irq_enable+0x4/0x8
[ 176.198960] cpuidle_enter+0x34/0x48
[ 176.202547] do_idle+0x1c4/0x234
[ 176.205786] cpu_startup_entry+0x30/0x34
[ 176.209723] secondary_start_kernel+0x124/0x144
[ 176.214267] __secondary_switched+0x6c/0x70
[ 176.218458] ---[ end trace 0000000000000000 ]---
Or, do you mean to add a uclamp_rq_dec() instead? But even if I skip the uclamp_rq_inc() at that line (which is equivalent to doing uclamp_rq_dec() after enqueue_task()), I still get a WARN like this:
Juno-r2-Debian login: [ 77.183511] ------------[ cut here ]------------
[ 77.188157] WARNING: CPU: 2 PID: 114 at kernel/sched/core.c:1594 uclamp_rq_inc+0x188/0x1a0
[ 77.196455] Modules linked in:
[ 77.199519] CPU: 2 PID: 114 Comm: kworker/u25:3 Not tainted 6.10.0-rc1-ga946a2b48119-dirty #1035
[ 77.208325] Hardware name: ARM Juno development board (r2) (DT)
[ 77.214255] Workqueue: rpciod rpc_async_schedule
[ 77.218893] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 77.225873] pc : uclamp_rq_inc+0x188/0x1a0
[ 77.229979] lr : enqueue_task+0x44/0x7c
[ 77.233826] sp : ffffffc082af3c80
[ 77.237143] x29: ffffffc082af3c80 x28: ffffffc081cd6000 x27: 0000000000000402
[ 77.244305] x26: ffffffc8fd285000 x25: ffffffc081cbaf40 x24: ffffff8800f92300
[ 77.251466] x23: ffffff897ef3ff40 x22: 00000000000000c0 x21: ffffff897ef3ff40
[ 77.258625] x20: ffffff88028f3480 x19: 0000000000000009 x18: 0000000000000000
[ 77.265784] x17: 0000000000000000 x16: 0000000000000001 x15: 00000000000003ae
[ 77.272943] x14: 000000000003eca5 x13: 000000000003eca5 x12: 00000000000000eb
[ 77.280102] x11: 0000000000000000 x10: 0000000000001800 x9 : 0000000000000000
[ 77.287261] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 000000000000003f
[ 77.294420] x5 : 0000000000000311 x4 : 0000000000000113 x3 : 000000000000017f
[ 77.301578] x2 : 0000000000000009 x1 : ffffff88028f3480 x0 : ffffff897ef3ff40
[ 77.308738] Call trace:
[ 77.311185] uclamp_rq_inc+0x188/0x1a0
[ 77.314944] activate_task+0x48/0x60
[ 77.318530] ttwu_do_activate+0x80/0x22c
[ 77.322464] try_to_wake_up+0x1e8/0x50c
[ 77.326311] wake_up_process+0x14/0x1c
[ 77.330070] kick_pool+0xb8/0x144
[ 77.333395] process_one_work+0x104/0x3a4
[ 77.337418] worker_thread+0x2b8/0x424
[ 77.341176] kthread+0xf8/0x194
[ 77.344327] ret_from_fork+0x10/0x20
[ 77.347913] ---[ end trace 0000000000000000 ]---
I pasted the trace here in case useful.
Hongyan
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 34 lines · Peter Zijlstra
Quoted 14 lines · Luis Machado
Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp behavior with the eevdf complete series applied.
Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing of tasks, and the uclamp values are organized in buckets.
Just for fun I added a few trace_printk's in
uclamp_eff_value,uclamp_rq_inc_idanduclamp_rq_dec_id.Booting up the system with
delayed_dequeuedisabled and running the benchmark, I see the uclamp bucket management pretty stable. Tasks get added to the uclamp buckets but then get removed. At the end of the benchmark, the uclamp buckets are (almost always) clean of tasks.Quoted 7 lines
Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50 tasks still. If I do another run, I can see 80, 100 tasks still.
I suspect refcounting might be going wrong somewhere due to delayed dequeue tasks, but that's more of a guess right now. Hopefully that is useful information. I'll resume investigation tomorrow.
Thank you both!!
Does the below help?
Note how
dequeue_task()doesuclamp_rq_dec()unconditionally, which is then not balanced in the case below.
kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3670
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str/* mustn't run a delayed task */ SCHED_WARN_ON (task_on_cpu (rq, p));enqueue_task (rq, p,ENQUEUE_DELAYED ); +uclamp_rq_inc (rq, p); }if (!task_on_cpu (rq, p)) {/*
As Hongyan pointed out in a separate message, the above makes things worse, as we end up with even more leftover tasks in the uclamp buckets.
I'm trying a fix in kernel/sched/core.c:enqueue_task that only calls uclamp_rq_inc if the task is not sched_delayed, so:
- uclamp_rq_inc(rq, p);
+ if (!p->se.sched_delayed)
+ uclamp_rq_inc(rq, p);
I'm not entirely sure it is correct, but it seems to fix things, but I'm still running some tests.
With the current code, given uclamp_rq_inc and uclamp_rq_dec get called in enqueue_task and dequeue_task, the additional enqueue_task call from ttwu_runnable for a delayed_dequeue task may do an additional unconditional call to uclamp_rq_inc, no?
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 54 lines · Luis Machado
Quoted 34 lines · Peter Zijlstra
Quoted 14 lines · Luis Machado
Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp behavior with the eevdf complete series applied.
Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing of tasks, and the uclamp values are organized in buckets.
Just for fun I added a few trace_printk's in
uclamp_eff_value,uclamp_rq_inc_idanduclamp_rq_dec_id.Booting up the system with
delayed_dequeuedisabled and running the benchmark, I see the uclamp bucket management pretty stable. Tasks get added to the uclamp buckets but then get removed. At the end of the benchmark, the uclamp buckets are (almost always) clean of tasks.Quoted 7 lines
Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50 tasks still. If I do another run, I can see 80, 100 tasks still.
I suspect refcounting might be going wrong somewhere due to delayed dequeue tasks, but that's more of a guess right now. Hopefully that is useful information. I'll resume investigation tomorrow.
Thank you both!!
Does the below help?
Note how
dequeue_task()doesuclamp_rq_dec()unconditionally, which is then not balanced in the case below.kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3670
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str/* mustn't run a delayed task */ SCHED_WARN_ON (task_on_cpu (rq, p));enqueue_task (rq, p,ENQUEUE_DELAYED ); +uclamp_rq_inc (rq, p); }if (!task_on_cpu (rq, p)) {/* As Hongyan pointed out in a separate message, the above makes things worse, as we end up with even more leftover tasks in the uclamp buckets.
I'm trying a fix in kernel/sched/core.c:enqueue_task that only calls
uclamp_rq_incif the task is notsched_delayed, so:(quoted diff) +2
-1- uclamp_rq_inc(rq, p); + if (!p->se.sched_delayed) + uclamp_rq_inc(rq, p);
Quoted 2 lines
I'm not entirely sure it is correct, but it seems to fix things, but I'm still running some tests.
This seems to so far not trigger any WARN or bad behavior in my tests.
Quoted 5 lines
With the current code, given
uclamp_rq_incanduclamp_rq_decget called inenqueue_taskanddequeue_task, the additionalenqueue_taskcall fromttwu_runnablefor adelayed_dequeuetask may do an additional unconditional call touclamp_rq_inc, no?
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Luis MachadoQuoted 27 lines · Luis Machado
kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3670
Quoted 13 lines · Peter Zijlstra
Note how
dequeue_task()doesuclamp_rq_dec()unconditionally, which is then not balanced in the case below.kernel/sched/core.c +1
-0@@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str/* mustn't run a delayed task */ SCHED_WARN_ON (task_on_cpu (rq, p));enqueue_task (rq, p,ENQUEUE_DELAYED ); +uclamp_rq_inc (rq, p); }if (!task_on_cpu (rq, p)) {/* As Hongyan pointed out in a separate message, the above makes things worse, as we end up with even more leftover tasks in the uclamp buckets.
I'm trying a fix in kernel/sched/core.c:enqueue_task that only calls
uclamp_rq_incif the task is notsched_delayed, so:(quoted diff) +2
-1- uclamp_rq_inc(rq, p); + if (!p->se.sched_delayed) + uclamp_rq_inc(rq, p);
Quoted 7 lines
I'm not entirely sure it is correct, but it seems to fix things, but I'm still running some tests.
With the current code, given
uclamp_rq_incanduclamp_rq_decget called inenqueue_taskanddequeue_task, the additionalenqueue_taskcall fromttwu_runnablefor adelayed_dequeuetask may do an additional unconditional call touclamp_rq_inc, no?
Yes, I got enqueue_task() and class->enqueue_task() confused this morning.
But with the above, you skip inc for sched_delayed, but dequeue_task() will have done the dec, so isn't it then still unbalanced?
Oh well, I'll go stare at this in tomorrow.
In any case, is there a uclamp self-test somewhere?
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 4 lines · Peter Zijlstra
But with the above, you skip inc for
sched_delayed, butdequeue_task()will have done the dec, so isn't it then still unbalanced?Oh well, I'll go stare at this in tomorrow.
OK, just before I went to play in my giant hamster wheel it hit me.
When dequeue_task() 'fails' and sets sched_delayed, we'll have done uclamp_rq_dec().
Then, since the delayed task is still on the rq -- per the failure -- it can be migrated, this will again do dequeue_task() which will *agian* do a uclamp_rq_dec().
So now we have a double dequeue -- *FAIL*.
Worse, the migration will then do an enqueue_task() on the new rq causing uclamp_rq_inc(). If you then get a ttwu() / ENQUEUE_DELAYED, you can tickle yet another uclamp_rq_inc() for another fail.
Something like the below avoids uclamp_rq_{inc,dec}() when ->sched_delayed, and moves it post class->enqueue_task() such that for the ENQUEUE_DELAYED case, we *will* do the inc after ->sched_delayed gets cleared.
Hmm?
---
kernel/sched/core.c +9 -1
@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);
@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }
@@ -1979,9 +1985,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +uclamp_rq_inc (rq, p); +if (sched_core_enabled (rq))sched_core_enqueue (rq, p); }
@@ -2003,6 +2010,7 @@ bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) }uclamp_rq_dec (rq, p); +return p->sched_class->dequeue_task (rq, p, flags); }
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
But with the above, you skip inc for
sched_delayed, butdequeue_task()will have done the dec, so isn't it then still unbalanced?Oh well, I'll go stare at this in tomorrow.
It did not appear to keep things unbalanced towards uclamp_rq_dec, but might've been luck of not hitting the right code path.
In any case ...
Quoted 61 lines
OK, just before I went to play in my giant hamster wheel it hit me.
When
dequeue_task()'fails' and setssched_delayed, we'll have doneuclamp_rq_dec().Then, since the delayed task is still on the rq -- per the failure -- it can be migrated, this will again do
dequeue_task()which will *agian* do auclamp_rq_dec().So now we have a double dequeue -- *FAIL*.
Worse, the migration will then do an
enqueue_task()on the new rq causinguclamp_rq_inc(). If you then get attwu()/ENQUEUE_DELAYED, you can tickle yet anotheruclamp_rq_inc()for another fail.Something like the below avoids uclamp_rq_{inc,dec}() when ->sched_delayed, and moves it post
class->enqueue_task()such that for theENQUEUE_DELAYEDcase, we *will* do the inc after ->sched_delayed gets cleared.Hmm?
---
kernel/sched/core.c +8
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,9 +1985,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +uclamp_rq_inc (rq, p); +if (sched_core_enabled (rq))sched_core_enqueue (rq, p); }
kernel/sched/core.c bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2016
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -2003,6 +2010,7 @@ bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) }uclamp_rq_dec (rq, p); +return p->sched_class->dequeue_task (rq, p, flags); }
... thanks for the patch! The above seems to do it for me. I can see more reasonable energy use with the eevdf-complete series. Still a bit higher. Might be noise, we'll see.
I'll go stare at it and run some more tests on our end with this fix applied on top and will report back.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 3 lines · Luis Machado
... thanks for the patch! The above seems to do it for me. I can see more reasonable energy use with the eevdf-complete series. Still a bit higher. Might be noise, we'll see.
W00t!!!
Let me write a decent Changelog and stuff it in the git tree along with all the other bits.
Thanks for all the testing.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Luis Machado
... thanks for the patch! The above seems to do it for me. I can see more reasonable energy use with the eevdf-complete series. Still a bit higher. Might be noise, we'll see.
W00t!!!
Let me write a decent Changelog and stuff it in the git tree along with all the other bits.
Thanks for all the testing.
I've been doing some more testing of the eevdf-complete series with the Pixel6/EAS platform. Hopefully these numbers will prove useful.
The energy regression from the original delayed-dequeue patch seems to have been mostly fixed (with your proposed patch). Energy readings for the big and mid cores are mostly stable and comparable with stock eevdf (without eevdf-complete).
The only difference I can spot now is in the energy use of the little cores. Compared to stock eevdf, the delayed-dequeue code seems to make the energy use of the little cores a bit spiky, meaning sometimes we get the expected level of energy use, but other times we get 40%, 60% or even 90% more.
For instance...
(1) m6.6-eevdf-stock*: stock eevdf runs
(2) m6.6-eevdf-complete-ndd-dz*: eevdf-complete + NO_DELAY_DEQUEUE + DELAY_ZERO
(3) m6.6-eevdf-complete-dd-dz*: eevdf-complete + DELAY_DEQUEUE + DELAY_ZERO
+------------+---------------------------------+-----------+ | channel | tag | perc_diff | +------------+---------------------------------+-----------+ | CPU-Little | m6.6-eevdf-stock-1 | 0.0% | | CPU-Little | m6.6-eevdf-stock-2 | -4.21% | | CPU-Little | m6.6-eevdf-stock-3 | -7.86% | | CPU-Little | m6.6-eevdf-stock-4 | -5.67% | | CPU-Little | m6.6-eevdf-stock-5 | -6.61% | | CPU-Little | m6.6-eevdf-complete-ndd-dz-1 | -2.21% | | CPU-Little | m6.6-eevdf-complete-ndd-dz-2 | -9.99% | | CPU-Little | m6.6-eevdf-complete-ndd-dz-3 | -6.1% | | CPU-Little | m6.6-eevdf-complete-ndd-dz-4 | -5.66% | | CPU-Little | m6.6-eevdf-complete-ndd-dz-5 | -7.12% | | CPU-Little | m6.6-eevdf-complete-dd-dz-1 | 96.69% | | CPU-Little | m6.6-eevdf-complete-dd-dz-2 | 22.1% | | CPU-Little | m6.6-eevdf-complete-dd-dz-3 | 44.82% | | CPU-Little | m6.6-eevdf-complete-dd-dz-4 | -0.23% | | CPU-Little | m6.6-eevdf-complete-dd-dz-5 | 8.28% | +------------+---------------------------------+-----------+
Looking at what might explain the spiky behavior with DELAY_DEQUEUE, I noticed the idle residency data (we have 2 idle states) also shows some spikyness and potential clues.
Looks like (1) and (2) manage to switch to idle states in a consistent manner, whereas (3) seems a bit erratic and more prone to take a shallower idle state (idle 0) as opposed to a deeper idle state (idle 1).
(1) and (2) seem to make better use of the deeper idle state.
+-------------------------------+---------+------------+-------+ | tag | cluster | idle_state | time | +-------------------------------+---------+------------+-------+ | m6.6-eevdf-stock-1 | little | not idle | 63.49 | | m6.6-eevdf-stock-1 | little | idle 0 | 30.66 | | m6.6-eevdf-stock-1 | little | idle 1 | 12.15 | | m6.6-eevdf-stock-2 | little | not idle | 62.6 | | m6.6-eevdf-stock-2 | little | idle 0 | 31.13 | | m6.6-eevdf-stock-2 | little | idle 1 | 14.56 | | m6.6-eevdf-stock-3 | little | not idle | 63.98 | | m6.6-eevdf-stock-3 | little | idle 0 | 31.54 | | m6.6-eevdf-stock-3 | little | idle 1 | 15.91 | | m6.6-eevdf-stock-4 | little | not idle | 64.18 | | m6.6-eevdf-stock-4 | little | idle 0 | 31.32 | | m6.6-eevdf-stock-4 | little | idle 1 | 15.83 | | m6.6-eevdf-stock-5 | little | not idle | 63.32 | | m6.6-eevdf-stock-5 | little | idle 0 | 30.4 | | m6.6-eevdf-stock-5 | little | idle 1 | 14.33 | | m6.6-eevdf-complete-ndd-dz-1 | little | not idle | 62.62 | | m6.6-eevdf-complete-ndd-dz-1 | little | idle 0 | 29.48 | | m6.6-eevdf-complete-ndd-dz-1 | little | idle 1 | 13.19 | | m6.6-eevdf-complete-ndd-dz-2 | little | not idle | 64.12 | | m6.6-eevdf-complete-ndd-dz-2 | little | idle 0 | 27.62 | | m6.6-eevdf-complete-ndd-dz-2 | little | idle 1 | 14.73 | | m6.6-eevdf-complete-ndd-dz-3 | little | not idle | 62.86 | | m6.6-eevdf-complete-ndd-dz-3 | little | idle 0 | 27.87 | | m6.6-eevdf-complete-ndd-dz-3 | little | idle 1 | 14.97 | | m6.6-eevdf-complete-ndd-dz-4 | little | not idle | 63.01 | | m6.6-eevdf-complete-ndd-dz-4 | little | idle 0 | 28.2 | | m6.6-eevdf-complete-ndd-dz-4 | little | idle 1 | 14.11 | | m6.6-eevdf-complete-ndd-dz-5 | little | not idle | 62.1 | | m6.6-eevdf-complete-ndd-dz-5 | little | idle 0 | 29.06 | | m6.6-eevdf-complete-ndd-dz-5 | little | idle 1 | 14.73 | | m6.6-eevdf-complete-dd-dz-1 | little | not idle | 46.18 | | m6.6-eevdf-complete-dd-dz-1 | little | idle 0 | 53.78 | | m6.6-eevdf-complete-dd-dz-1 | little | idle 1 | 3.75 | | m6.6-eevdf-complete-dd-dz-2 | little | not idle | 57.64 | | m6.6-eevdf-complete-dd-dz-2 | little | idle 0 | 40.47 | | m6.6-eevdf-complete-dd-dz-2 | little | idle 1 | 7.39 | | m6.6-eevdf-complete-dd-dz-3 | little | not idle | 43.14 | | m6.6-eevdf-complete-dd-dz-3 | little | idle 0 | 57.73 | | m6.6-eevdf-complete-dd-dz-3 | little | idle 1 | 3.65 | | m6.6-eevdf-complete-dd-dz-4 | little | not idle | 58.97 | | m6.6-eevdf-complete-dd-dz-4 | little | idle 0 | 36.4 | | m6.6-eevdf-complete-dd-dz-4 | little | idle 1 | 9.42 | | m6.6-eevdf-complete-dd-dz-5 | little | not idle | 55.85 | | m6.6-eevdf-complete-dd-dz-5 | little | idle 0 | 36.75 | | m6.6-eevdf-complete-dd-dz-5 | little | idle 1 | 13.14 | +-------------------------------+---------+------------+-------+
I can't draw a precise conclusion, but it might be down to delayed util_est updates or even the additional time the delayed-dequeue tasks spend on the runqueue. But delayed-dequeue does change the overall behavior a bit on these heterogeneous platforms, energy-wise.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Luis MachadoQuoted 13 lines · Luis Machado
(2) m6.6-eevdf-complete: m6.6-stock plus this series.
(3) m6.6-eevdf-complete-no-delay-dequeue: (2) +NO_DELAY_DEQUEUE+------------+------------------------------------------------------+-----------+ | cluster | tag | perc_diff | +------------+------------------------------------------------------+-----------+ | CPU | m6.6-stock | 0.0% | | CPU-Big | m6.6-stock | 0.0% | | CPU-Little | m6.6-stock | 0.0% | | CPU-Mid | m6.6-stock | 0.0% | | GPU | m6.6-stock | 0.0% | | Total | m6.6-stock | 0.0% |
Quoted 6 lines
| CPU | m6.6-eevdf-complete-no-delay-dequeue | 117.77% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue | 113.79% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue | 97.47% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue | 189.0% | | GPU | m6.6-eevdf-complete-no-delay-dequeue | -6.74% | | Total | m6.6-eevdf-complete-no-delay-dequeue | 103.84% |
This one is still flummoxing me. I've gone over the patch a few times on different days and I'm not seeing it. Without DELAY_DEQUEUE it should behave as before.
Let me try and split this patch up into smaller parts such that you can try and bisect this.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Peter,
Quoted 22 lines · Peter Zijlstra
Quoted 13 lines · Luis Machado
(2) m6.6-eevdf-complete: m6.6-stock plus this series.
(3) m6.6-eevdf-complete-no-delay-dequeue: (2) +NO_DELAY_DEQUEUE+------------+------------------------------------------------------+-----------+ | cluster | tag | perc_diff | +------------+------------------------------------------------------+-----------+ | CPU | m6.6-stock | 0.0% | | CPU-Big | m6.6-stock | 0.0% | | CPU-Little | m6.6-stock | 0.0% | | CPU-Mid | m6.6-stock | 0.0% | | GPU | m6.6-stock | 0.0% | | Total | m6.6-stock | 0.0% |Quoted 6 lines
| CPU | m6.6-eevdf-complete-no-delay-dequeue | 117.77% | | CPU-Big | m6.6-eevdf-complete-no-delay-dequeue | 113.79% | | CPU-Little | m6.6-eevdf-complete-no-delay-dequeue | 97.47% | | CPU-Mid | m6.6-eevdf-complete-no-delay-dequeue | 189.0% | | GPU | m6.6-eevdf-complete-no-delay-dequeue | -6.74% | | Total | m6.6-eevdf-complete-no-delay-dequeue | 103.84% |
Quoted 7 lines
This one is still flummoxing me. I've gone over the patch a few times on different days and I'm not seeing it. Without
DELAY_DEQUEUEit should behave as before.Let me try and split this patch up into smaller parts such that you can try and bisect this.
Same situation on my end. I've been chasing this for some time and I don't fully understand why things go off the rails energy-wise as soon as DELAY_DEQUEUE is enabled, now that the load_avg accounting red herring is gone.
I do have one additional piece of information though. Hopefully it will be useful.
Booting the kernel with NO_DELAY_DEQUEUE (default to false), things work fine. Then if I switch to DELAY_DEQUEUE at runtime, things start using a lot more power.
The interesting bit is if I switch to NO_DELAY_DEQUEUE again at runtime, things don't go back to normal. Rather they stay the same, using a lot more energy.
I wonder if we're leaving some unbalanced state somewhere while DELAY_DEQUEUE is on, something that is signalling we have more load/utilization than we actually do.
The PELT signals look reasonable from what I can see. We don't seem to be boosting frequencies, but we're running things mostly on big cores with DELAY_DEQUEUE on.
I'll keep investigating this. Please let me know if you need some additional data or testing and I can get that going.
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 5 lines · Luis Machado
Booting the kernel with
NO_DELAY_DEQUEUE(default to false), things work fine. Then if I switch toDELAY_DEQUEUEat runtime, things start using a lot more power.The interesting bit is if I switch to
NO_DELAY_DEQUEUEagain at runtime, things don't go back to normal. Rather they stay the same, using a lot more energy.
Ooh, cute.. weird. I'll try and see if we leak state somehow.
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Luis Machado
Booting the kernel with
NO_DELAY_DEQUEUE(default to false), things work fine. Then if I switch toDELAY_DEQUEUEat runtime, things start using a lot more power.The interesting bit is if I switch to
NO_DELAY_DEQUEUEagain at runtime, things don't go back to normal. Rather they stay the same, using a lot more energy.Ooh, cute.. weird. I'll try and see if we leak state somehow.
Hi. I'm working on uclamp anyway so I just gave this series (and all fixes mentioned in this thread) a go.
It seems after running with this series for a while, suddenly uclamp_max no longer works. Running a task with uclamp_max of 110 still sees an rq uclamp_max of 1024. Also weirdly, the PELT signal of that uclamp_max is completely broken. It goes up to a very high value and then suddenly drops to almost 0 from time to time.
I wonder if the reason Luis saw that is because he was the first one to run delayed dequeue with uclamp. I'm not familiar enough with the series to know how it could possibly affect uclamp so investigating in this front may be worth trying. Maybe some leaked uclamp state on the rq?
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 9 lines · Peter Zijlstra
Quoted 7 lines · Luis Machado
Quoted 6 lines
Bisecting through the patches in this series, I ended up with patch 08/10 as the one that improved things overall for these benchmarks.
I'd like to investigate this further to understand the reason behind some of these dramatic improvements.
Quoted 7 lines
Quoted 4 lines
Investigating these improvements a bit more, I noticed they came with a significantly higher power usage on the Pixel6 (where EAS is enabled). I bisected it down to the delayed dequeue patch. Disabling
DELAY_DEQUEUEandDELAY_ZEROat runtime doesn't help in bringing the power usage down.Hmm, that is unexpected. The intent was for
NO_DELAY_DEQUEUEto fully disable things. I'll go have a prod at it.
I'm running a few more numbers to confirm this situation with the feature switches.
Quoted 9 lines
Quoted 6 lines
Though I don't fully understand the reason behind this change in behavior yet, I did spot the benchmark processes running almost entirely on the big core cluster, with little to no use of the little core and mid core clusters.
That would explain higher power usage and also the significant jump in performance.
ISTR you (arm) has these tools to trace and plot the varioud util values. This should be readily reflected there if that is the case, no?
Indeed we do, but I'm still in the process of compiling those numbers into a meaningful plot, so I'm afraid I don't have those handy yet, sorry.
Quoted 10 lines
Quoted 4 lines
I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of utilization/load of the runqueue and, as a consequence, we're scheduling things to run on higher OPP's in the big cores, leading to poor decisions for energy efficiency.
Notably
util_est_update()gets delayed. Given we don't actually do an enqueue when a delayed task gets woken, it didn't seem to make sense to update that sooner.I'll go over all that again
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter Zijlstra(removes apparently busted bytedance.com address and retries xmit)
Greetings!
With this version, the better CPU distribution (for tbench synchronous net-blasting) closed the CFS vs EEVDF throughput deficit. I verified both by rolling the previous version forward and back-porting to 6.1 where I've got CFS and EEVDF to re-compare, now with both dequeue delay patch versions.
As usual, there will be winners and losers, but (modulo dead buglet) it looks kinda promising to me.
Distribution of single pinned buddy pair measured in master:DELAY_DEQUEUE
----------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
----------------------------------------------------------------------------------------------------------
tbench:(2) | 6277.099 ms | 1597104 | avg: 0.003 ms | max: 0.129 ms | sum: 4334.723 ms | tbench_srv:(2) | 5724.971 ms | 1682629 | avg: 0.001 ms | max: 0.083 ms | sum: 2076.616 ms |
----------------------------------------------------------------------------------------------------------
TOTAL: | 12021.128 ms | 3280275 | | 1.729 ms | 6425.483 ms |
----------------------------------------------------------------------------------------------------------
client/server CPU distribution ~52%/48%
NO_DELAY_DEQUEUE
----------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
----------------------------------------------------------------------------------------------------------
tbench:(2) | 6724.774 ms | 1546761 | avg: 0.002 ms | max: 0.409 ms | sum: 2443.549 ms | tbench_srv:(2) | 5275.329 ms | 1571688 | avg: 0.002 ms | max: 0.086 ms | sum: 2734.151 ms |
----------------------------------------------------------------------------------------------------------
TOTAL: | 12019.641 ms | 3119000 | | 9996.367 ms | 15187.833 ms |
----------------------------------------------------------------------------------------------------------
client/server CPU distribution ~56%/44%
Note switches and delay sum. For tbench, they translate directly to throughput. The other shoe lands with async CPU hog net-blasters, for those, scheduler cycles tends to be wasted cycles.
-Mike
Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Quoted 7 lines
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Since we should have dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
I stumbled upon a subtle change between CFS and EEVDF which kinda relates to the question how long lag should be carried around, but on the other side of the 0-lag value.
Assume that we have two tasks taking turns on a single CPU. Task 1 does something and wakes up Task 2.
Task 2 does something and goes to sleep.
And we're just repeating that.
Task 1 and task 2 only run for very short amounts of time, i.e. much shorter than a regular time slice.
Let's further assume, that task 1 runs longer than task 2.
In CFS, this means, that vruntime of task 1 starts to outrun the vruntime of task 2. This means that vruntime(task2) < vruntime(task1). Hence, task 2 always gets picked on wake up because it has the smaller vruntime. In EEVDF, this would translate to a permanent positive lag, which also causes task 2 to get consistently scheduled on wake up.
Let's now assume, that ocassionally, task 2 runs a little bit longer than task 1. In CFS, this means, that task 2 can close the vruntime gap by a bit, but, it can easily remain below the value of task 1. Task 2 would still get picked on wake up.
With EEVDF, in its current form, task 2 will now get a negative lag, which in turn, will cause it not being picked on the next wake up.
So, it seems we have a change in the level of how far the both variants look into the past. CFS being willing to take more history into account, whereas EEVDF does not (with update_entity_lag setting the lag value from scratch, and place_entity not taking the original vruntime into account).
All of this can be seen as correct by design, a task consumes more time than the others, so it has to give way to others. The big difference is now, that CFS allowed a task to collect some bonus by constantly using less CPU time than others and trading that time against ocassionally taking more CPU time. EEVDF could do the same thing, by allowing the accumulation of (positive) lag, which can then be traded against the one time the task would get negative lag.
This might of course clash with other EEVDF assumptions but blends with the question on how long lag should be preserved.
I stumbled upon a performance degredation caused by this nuance. EEVDF is not necessarily at fault here. The problem rather lies in a component that relies on CFS allowing the above mentioned bonus aggregation while knowing that the corresponding task 2 has to run.
See: https://lore.kernel.org/netdev/ZWbapeL34Z8AMR5f@DESKTOP-2CCOB1S./T/
With the reasoning above, the degradation can be explained perfectly, and even be fixed by allowing the accumulation of lag by applying the patch below.
I was just wondering if my assumptions here are correct and whether this would be an implicit change that one should be aware about.
kernel/sched/fair.c +1 -1
@@ -701,7 +701,7 @@ static void update_entity_lag(struct cfs_rq *cfs_rq, struct sched_entity *se)s64 lag, limit;SCHED_WARN_ON (!se->on_rq); - lag =avg_vruntime (cfs_rq) - se->vruntime; + lag = se->vlag +avg_vruntime (cfs_rq) - se->vruntime; limit =calc_delta_fair (max_t (u64,2 *se->slice,TICK_NSEC ), se); se->vlag =clamp (lag, -limit, limit);
A message that is not in the archive (sent off-list or not yet received): [email protected]
Peter Zijlstra
maintainerRe: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Quoted 6 lines · Libo Chen
Hi Peter,
We are observing massive hackbench regressions with this patchset as well as the newer version from your queue. I don’t know if you are still working on complete EEVDF, just want to bring this issue to your attention if it’s still in play.
Yeha, I'm still (slowly) poking at this. I was planning to post a new set in a week or so.
Quoted 9 lines
The test system is a two-socket Zen4 EYPC with total of 192C/384T, intel does suffer too but to a lesser degree. Kernel is v6.10 based off the latest schedule/core. From 48 groups to 296 groups, process/threads via socket degrades <= 60%, process/threads via pipe can have regression at whopping 110% or more.
It turns out in
deactivate_task(), this patchset changes the order ofdequeue_task()and settingp->on_rqsuch thatp->on_rqis set afterdequeue_task()gets called.
Bah, I knew I had to double check that,.. but then I never did :/
Quoted 10 lines
My understanding is
TASK_ON_RQ_MIGRATING
allows src rq lock to be released to other tasks since soon-to-be-dequeued task is migrating anyway, so holding two rq locks while dequeuing seems to be a quite horrible thing to do. The RFC patch is a bit tricky to get the order back, but a fix can be easily done to your newer version specifically to “sched: SplitDEQUEUE_SLEEPfrom deactivate_task()” like below:void
deactivate_task(struct rq *rq, struct task_struct *p, int flags){
Review
Quoted 7 lines
(quoted diff) +2
-2- dequeue_task(rq, p, flags); - WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); ASSERT_EXCLUSIVE_WRITER(p->on_rq); + + dequeue_task(rq, p, flags); }
Thanks, I'll fold that in and try and put your name somewhere.
Peter Zijlstra
maintainerRFC PATCH 9/10 sched/eevdf: Allow shorter slices to wakeup-preempt
In reply to Peter ZijlstraPart of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT
1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |
100ms massive_intr 500us cyclictest PREEMPT_SHORT
1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.
--- kernel/sched/fair.c | 11 ++++++++--- kernel/sched/features.h | 4 ++++ 2 files changed, 12 insertions(+), 3 deletions(-)
kernel/sched/fair.c +8 -3
@@ -8538,9 +8538,14 @@ static void check_preempt_wakeup_fair(st cfs_rq =cfs_rq_of (se);update_curr (cfs_rq); -/* -* XXX pick_eevdf(cfs_rq) != se ? -*/ +if (sched_feat (PREEMPT_SHORT ) && pse->slice < se->slice && +entity_eligible (cfs_rq, pse) && + (s64)(pse->deadline - se->deadline) <0 && + se->vlag == se->deadline) { +/* negate RUN_TO_PARITY */ + se->vlag = se->deadline -1 ; + } +if (pick_eevdf (cfs_rq) == pse) goto preempt;
kernel/sched/features.h +4 -0
@@ -14,6 +14,10 @@ SCHED_FEAT(PLACE_DEADLINE_INITIAL, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow tasks with a shorter slice to disregard RUN_TO_PARITY + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Peter Zijlstra
maintainerRFC PATCH 10/10 sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to Peter ZijlstraAllow applications to directly set a suggested request/slice length using sched_attr::sched_runtime.
The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
For all the following examples assume a scheduling quantum of 8, and for consistency all examples have W=4:
{A,B,C,D}(w=1,r=8):
ABCD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+*------+-------+--- ---+--*----+-------+---
t=2, V=5.5 t=3, V=7.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+----*--+-------+--- ---+------*+-------+---
Note: 4 identical tasks in FIFO order
~~~
{A,B}(w=1,r=16) C(w=2,r=16)
AACCBBCC...
+---+---+---+---
t=0, V=1.25 t=2, V=5.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=4, V=8.25 t=6, V=12.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+-------*-------+--- ---+-------+---*---+---
Note: 1 heavy task -- because q=8, double r such that the deadline of the w=2
task doesn't go below q.
Note: observe the full schedule becomes: W*max(r_i/w_i) = 4*2q = 8q in length.
Note: the period of the heavy task is half the full period at:
W*(r_i/w_i) = 4*(2q/2) = 4q
~~~
{A,C,D}(w=1,r=16) B(w=1,r=8):
BAACCBDD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |--------------< A |---------------<
B |------< B |------<
C |--------------< C |--------------<
D |--------------< D |--------------<
---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.5 t=5, V=11.5
A |---------------< A |---------------< B |------< B |------< C |--------------< C |--------------< D |--------------< D |--------------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.5
A |---------------<
B |------<
C |--------------<
D |--------------<
---+-------+----*--+---
Note: 1 short task -- again double r so that the deadline of the short task
won't be below q. Made B short because its not the leftmost task, but is eligible with the 0,1,2,3 spread.
Note: like with the heavy task, the period of the short task observes:
W*(r_i/w_i) = 4*(1q/1) = 4q
~~~
A(w=1,r=16) B(w=1,r=8) C(w=2,r=16)
BCCAABCC...
+---+---+---+---
t=0, V=1.25 t=1, V=3.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.25 t=5, V=11.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.25
A |--------------<
B |------<
C |------<
---+-------+----*--+---
Note: 1 heavy and 1 short task -- combine them all.
Note: both the short and heavy task end up with a period of 4q
~~~
A(w=1,r=16) B(w=2,r=16) C(w=1,r=8)
BBCAABBC...
+---+---+---+---
t=0, V=1 t=2, V=5
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=3, V=7 t=5, V=11
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=7, V=15
A |--------------<
B |------<
C |------<
---+-------+------*+---
Note: as before but permuted
~~~
Quoted 1 line
From all this it can be deduced that, for the steady state:
- the total period (P) of a schedule is: W*max(r_i/w_i)
- the average period of a task is: W*(r_i/w_i)
- each task obtains the fair share: w_i/W of each full period P
--- include/linux/sched.h | 5 ++++- kernel/sched/core.c | 33 ++++++++++++++++++++++++++------- kernel/sched/debug.c | 3 ++- kernel/sched/fair.c | 6 ++++-- 4 files changed, 36 insertions(+), 11 deletions(-)
include/linux/sched.h +4 -1
@@ -542,7 +542,10 @@ struct sched_entity {struct list_head group_node;unsigned int on_rq; -unsigned int sched_delayed; + +unsigned char sched_delayed; +unsigned char custom_slice; +/* 2 byte hole */ u64 exec_start;u64 sum_exec_runtime;
kernel/sched/core.c +26 -7
@@ -4550,7 +4550,6 @@ static void __sched_fork(unsigned long c p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ; - p->se.slice = sysctl_sched_base_slice;INIT_LIST_HEAD (&p->se.group_node);#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -4801,6 +4800,8 @@ int sched_fork(unsigned long clone_flags p->prio = p->normal_prio = p->static_prio;set_load_weight (p, false); + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice;/* * We don't need the reset flag anymore after the fork. It has
@@ -7627,10 +7628,20 @@ static void __setscheduler_params(struct p->policy = policy; -if (dl_policy (policy)) +if (dl_policy (policy)) {__setparam_dl (p, attr); -else if (fair_policy (policy)) + }else if (fair_policy (policy)) { p->static_prio =NICE_TO_PRIO (attr->sched_nice); +if (attr->sched_runtime) { + p->se.custom_slice =1 ; + p->se.slice =clamp_t (u64, attr->sched_runtime, +NSEC_PER_MSEC /10 ,/* HZ=1000 * 10 */ +NSEC_PER_MSEC *100 );/* HZ=100 / 10 */ + }else { + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice; + } + }/* * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -7812,7 +7823,9 @@ static int __sched_setscheduler(struct t* but store a possible modification of reset_on_fork. */ if (unlikely (policy == p->policy)) { -if (fair_policy (policy) && attr->sched_nice !=task_nice (p)) +if (fair_policy (policy) && + (attr->sched_nice !=task_nice (p) || + (attr->sched_runtime && attr->sched_runtime != p->se.slice))) goto change;if (rt_policy (policy) && attr->sched_priority != p->rt_priority) goto change;
@@ -7958,6 +7971,9 @@ static int _sched_setscheduler(struct ta .sched_nice = PRIO_TO_NICE(p->static_prio), }; +if (p->se.custom_slice) + attr.sched_runtime = p->se.slice; +/* Fixup the legacy SCHED_RESET_ON_FORK hack. */ if ((policy !=SETPARAM_POLICY ) && (policy &SCHED_RESET_ON_FORK )) { attr.sched_flags |=SCHED_FLAG_RESET_ON_FORK ;
@@ -8124,12 +8140,14 @@ static int sched_copy_attr(struct sched_static void get_params (struct task_struct *p,struct sched_attr *attr) { -if (task_has_dl_policy (p)) +if (task_has_dl_policy (p)) {__getparam_dl (p, attr); -else if (task_has_rt_policy (p)) + }else if (task_has_rt_policy (p)) { attr->sched_priority = p->rt_priority; -else + }else { attr->sched_nice =task_nice (p); + attr->sched_runtime = p->se.slice; + } }/**
@@ -10084,6 +10102,7 @@ void __init sched_init(void) }set_load_weight (&init_task, false); + init_task.se.slice = sysctl_sched_base_slice,/* * The boot idle thread does lazy MMU switching as well:
kernel/sched/debug.c +2 -1
@@ -579,11 +579,12 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), SPLIT_NS(p->se.vruntime), entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), + p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice), SPLIT_NS(p->se.sum_exec_runtime), (long long)(p->nvcsw + p->nivcsw),
kernel/sched/fair.c +4 -2
@@ -984,7 +984,8 @@ static void update_deadline(struct cfs_r* nice) while the request time r_i is determined by * sysctl_sched_base_slice. */ - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice;/* * EEVDF: vd_i = ve_i + r_i / w_i
@@ -5169,7 +5170,8 @@ place_entity(struct cfs_rq *cfs_rq, struu64 vslice, vruntime =avg_vruntime (cfs_rq);s64 lag =0 ; - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se);/*
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
On Fri, 05 Apr 2024 12:28:04 +0200 Peter Zijlstra <[email protected]>
Quoted 6 lines
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Given HZ=100 for example, what is preventing applications of suggested slice=0.5ms from running 5ms a tick? If slice is 90ms otoh, is tick able to kick the curr that has been on cpu for 10ms off cpu, given cfs_rq->nr_running > 1?
Quoted 12 lines
kernel/sched/fair.c +2
-1@@ -984,7 +984,8 @@ static void update_deadline(struct cfs_r* nice) while the request time r_i is determined by * sysctl_sched_base_slice. */ - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice;/* * EEVDF: vd_i = ve_i + r_i / w_i
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to Peter Zijlstrakernel/sched/core.c static int __sched_setscheduler(struct t · line 7829
Quoted 12 lines · Peter Zijlstra
kernel/sched/core.c +3
-1@@ -7812,7 +7823,9 @@ static int __sched_setscheduler(struct t* but store a possible modification of reset_on_fork. */ if (unlikely(policy == p->policy)) { - if (fair_policy(policy) && attr->sched_nice != task_nice(p)) + if (fair_policy(policy) && + (attr->sched_nice != task_nice(p) || + (attr->sched_runtime && attr->sched_runtime != p->se.slice))) goto change;
Can we make that only look at attr->sched_runtime != p->se.slice? Doing so lets you clear a custom slice by.. clearing it.. rather than making tools rummage about for the proper value to overwrite.
-Mike
Peter Zijlstra
maintainerRe: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Quoted 17 lines · Mike Galbraith
kernel/sched/core.c static int __sched_setscheduler(struct t · line 7829
Quoted 12 lines · Peter Zijlstra
kernel/sched/core.c +3
-1@@ -7812,7 +7823,9 @@ static int __sched_setscheduler(struct t* but store a possible modification of reset_on_fork. */ if (unlikely(policy == p->policy)) { - if (fair_policy(policy) && attr->sched_nice != task_nice(p)) + if (fair_policy(policy) && + (attr->sched_nice != task_nice(p) || + (attr->sched_runtime && attr->sched_runtime != p->se.slice))) goto change;Can we make that only look at
attr->sched_runtime!=p->se.slice? Doing so lets you clear a custom slice by.. clearing it.. rather than making tools rummage about for the proper value to overwrite.
Duh yes. Seems I already did the right thing in __setscheduler_params() for that.
Done.
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Quoted 7 lines
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Is it possible to leverage this task slice to do better task wakeup placement? The idea is that, the smaller the slice the wakee has, the less idle CPU it should scan. This can reduce wake latency and inhibit costly task migration, especially on large systems.
We did some experiments and got some performance improvements:
From 9cb806476586d7048fcbd0f66d0101f0dbb8fd2b Mon Sep 17 00:00:00 2001
From: Chen Yu <[email protected]>
Date: Tue, 7 May 2024 22:36:29 +0800
Subject: [RFC PATCH] sched/eevdf: Use customized slice to reduce wakeup latency
and inhibit task migration
Problem 1:
The overhead of task migration is high on many-core system. The overhead brings performance penalty due to broken cache locality/higher cache-to-cache latency.
Problem 2:
During wakeup, the time spent on searching for an idle CPU is costly on many-core system. Besides, access to other CPU's rq statistics brings cace contention:
available_idle_cpu(cpu) -> idle_cpu(cpu) -> {rq->curr, rq->nr_running}
Although SIS_UTIL throttles the scan depth based on system utilization, there is requirement to further limit the scan depth for specific workload, especially for short duration wakee.
Now we have the interface to customize the request/slice. The smaller the slice is, the earlier the task can be picked up, and the lower wakeup latency the task expects. Leverage the wakee's slice to further throttle the idle CPU scan depth - the shorter slice, the less CPUs to scan.
Test on 240 CPUs, 2 sockets system. With SNC(sub-numa-cluster) enabled, each LLC domain has 60 CPUs. There is noticeable improvement of netperf. (With SNC disabled, more improvements should be seen because C2C is higher)
The global slice is 3 msec(sysctl_sched_base_slice) by default on my ubuntu 22.04, and the customized slice is set to 0.1 msec for both netperf and netserver:
for i in $(seq 1 $job); do
netperf_slice -e 100000 -4 -H 127.0.01 -t TCP_RR -c -C -l 100 &
done
case load baseline(std%) compare%( std%) TCP_RR 60-threads 1.00 ( 1.60) +0.35 ( 1.73) TCP_RR 120-threads 1.00 ( 1.34) -0.96 ( 1.24) TCP_RR 180-threads 1.00 ( 1.59) +92.20 ( 4.24) TCP_RR 240-threads 1.00 ( 9.71) +43.11 ( 2.97)
--- kernel/sched/fair.c | 23 ++++++++++++++++++++--- kernel/sched/features.h | 1 + 2 files changed, 21 insertions(+), 3 deletions(-)
kernel/sched/fair.c +20 -3
@@ -7368,6 +7368,24 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +/* +* Scale the scan number of idle CPUs according to customized +* wakee's slice. The smaller the slice is, the earlier the task +* wants be picked up, thus the lower wakeup latency the task expects. +* The baseline is the global sysctl_sched_base_slice. Task slice +* smaller than the global one would shrink the scan number. +*/ +static int adjust_idle_scan (struct task_struct *p,int nr) +{ +if (!sched_feat (SIS_FAST )) +return nr; + +if (!p->se.custom_slice || p->se.slice >= sysctl_sched_base_slice) +return nr; + +return div_u64 (nr * p->se.slice, sysctl_sched_base_slice); +} +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolif (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) { -/* because !--nr is the condition to stop scan */ - nr =READ_ONCE (sd_share->nr_idle_scan) +1 ; + nr =adjust_idle_scan (p,READ_ONCE (sd_share->nr_idle_scan));/* overloaded LLC is unlikely to have idle cpu/core */ -if (nr ==1 ) +if (nr <=0 )return -1 ; } }
kernel/sched/features.h +1 -0
@@ -50,6 +50,7 @@ SCHED_FEAT(TTWU_QUEUE, true)* When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SIS_FAST , true)/* * Issue a WARN when we do multiple update_rq_clock() calls
Signature · 1 line
2.25.1
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Quoted 12 lines · Chen Yu
Quoted 11 lines · Peter Zijlstra
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
Quoted 4 lines
Quoted 2 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Is it possible to leverage this task slice to do better task wakeup placement?
Slice being unrelated to placement makes its use in a placement related knob look wrong. Even the smallest possible slice is orders of magnitude larger than the cycle time of TCP_RR, making slice nearly irrelevant to the issue being demonstrating via TCP_RR. Even for that huge socket box, it won't take long as cycle time increases toward that smallest possible slice for the cost of needless wait to bury placement decision costs.
Quoted 3 lines
The idea is that, the smaller the slice the wakee has, the less idle CPU it should scan. This can reduce wake latency and inhibit costly task migration, especially on large systems.
Sure, this is an age old issue that's scaled up to size extra ugly in that huge socket box. Any solution needs to scale as well methinks, a simple fixed yardstick won't work, as the costs being mitigated vary wildly with platform size/shape.
-Mike
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Quoted 16 lines · Mike Galbraith
Quoted 15 lines · Chen Yu
Quoted 13 lines · Peter Zijlstra
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 6 lines
Quoted 1 line
Is it possible to leverage this task slice to do better task wakeup placement?
Slice being unrelated to placement makes its use in a placement related knob look wrong. Even the smallest possible slice is orders of magnitude larger than the cycle time of
TCP_RR, making slice nearly irrelevant to the issue being demonstrating viaTCP_RR.
Yes, I agree that there is no direct relationship between the slice and the task placement. The idea is to use slice as an input hint from the user to tell the kernel how much latency this user cares(but not expect the task to last for that long).
Quoted 4 lines
Even for that huge socket box, it won't take long as cycle time increases toward that smallest possible slice for the cost of needless wait to bury placement decision costs.
I see. The wake up latency is composed of:
1. The time to do placement decision.
2. The time wait in the runqueue to be picked.
Even if the 1 has been reduced, the 2 might take more time if the runqueue is busy. We can mitigate this by checking if there is <= 1 short duration task on that target CPU.
Quoted 9 lines
Quoted 3 lines
The idea is that, the smaller the slice the wakee has, the less idle CPU it should scan. This can reduce wake latency and inhibit costly task migration, especially on large systems.
Sure, this is an age old issue that's scaled up to size extra ugly in that huge socket box. Any solution needs to scale as well methinks, a simple fixed yardstick won't work, as the costs being mitigated vary wildly with platform size/shape.
Understand, this slice based placement was chosen for experimental purpose, and seek for directions. And I agree we should take the platform size(such as CPU number) into consideration.
thanks,
Chenyu
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Quoted 2 lines · Chen Yu
And I agree we should take the platform size(such
as CPU number) into consideration.
I think you'll need more that that, because size agnostic, when it comes to latency, an idle CPU is damn hard to beat, making migration restrictions (traditional and obvious target) tend to leave highly annoying piles of collateral damage in their wake.
(spoken from BTDT perspective, have t-shirt, got butt kicked;)
-Mike
K Prateek Nayak
reviewerRe: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to Chen YuHello Chenyu,
Quoted 104 lines · Chen Yu
Quoted 6 lines · Peter Zijlstra
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Quoted 7 lines
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Is it possible to leverage this task slice to do better task wakeup placement? The idea is that, the smaller the slice the wakee has, the less idle CPU it should scan. This can reduce wake latency and inhibit costly task migration, especially on large systems.
We did some experiments and got some performance improvements:
From
9cb806476586d7048fcbd0f66d0101f0dbb8fd2bMon Sep 17 00:00:00 2001
From: Chen Yu <[email protected]>
Date: Tue, 7 May 2024 22:36:29 +0800
Subject: [RFC PATCH] sched/eevdf: Use customized slice to reduce wakeup latency
and inhibit task migrationProblem 1:
The overhead of task migration is high on many-core system. The overhead brings performance penalty due to broken cache locality/higher cache-to-cache latency.Problem 2:
During wakeup, the time spent on searching for an idle CPU is costly on many-core system. Besides, access to other CPU's rq statistics brings cace contention:
available_idle_cpu(cpu)->idle_cpu(cpu)-> {rq->curr,rq->nr_running}Although
SIS_UTILthrottles the scan depth based on system utilization, there is requirement to further limit the scan depth for specific workload, especially for short duration wakee.Now we have the interface to customize the request/slice. The smaller the slice is, the earlier the task can be picked up, and the lower wakeup latency the task expects. Leverage the wakee's slice to further throttle the idle CPU scan depth - the shorter slice, the less CPUs to scan.
Test on 240 CPUs, 2 sockets system. With SNC(sub-numa-cluster) enabled, each LLC domain has 60 CPUs. There is noticeable improvement of netperf. (With SNC disabled, more improvements should be seen because C2C is higher)
The global slice is 3
msec(sysctl_sched_base_slice)by default on my ubuntu 22.04, and the customized slice is set to 0.1 msec for both netperf and netserver:for i in $(seq 1 $job); do
netperf_slice -e 100000 -4 -H 127.0.01 -t TCP_RR -c -C -l 100 &done
case load baseline(std%) compare%( std%) TCP_RR 60-threads 1.00 ( 1.60) +0.35 ( 1.73) TCP_RR 120-threads 1.00 ( 1.34) -0.96 ( 1.24) TCP_RR 180-threads 1.00 ( 1.59) +92.20 ( 4.24) TCP_RR 240-threads 1.00 ( 9.71) +43.11 ( 2.97)Suggested-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>--- kernel/sched/fair.c | 23 ++++++++++++++++++++--- kernel/sched/features.h | 1 + 2 files changed, 21 insertions(+), 3 deletions(-)kernel/sched/fair.c +18
-0@@ -7368,6 +7368,24 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +/* +* Scale the scan number of idle CPUs according to customized +* wakee's slice. The smaller the slice is, the earlier the task +* wants be picked up, thus the lower wakeup latency the task expects. +* The baseline is the global sysctl_sched_base_slice. Task slice +* smaller than the global one would shrink the scan number. +*/ +static int adjust_idle_scan (struct task_struct *p,int nr) +{ +if (!sched_feat (SIS_FAST )) +return nr; + +if (!p->se.custom_slice || p->se.slice >= sysctl_sched_base_slice) +return nr; + +return div_u64 (nr * p->se.slice, sysctl_sched_base_slice); +} +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7407
Quoted 8 lines
kernel/sched/fair.c +2
-2@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolif (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) { - /* because !--nr is the condition to stop scan */> - nr = READ_ONCE(sd_share->nr_idle_scan) + 1; + nr =adjust_idle_scan (p,READ_ONCE (sd_share->nr_idle_scan));/* overloaded LLC is unlikely to have idle cpu/core */ -if (nr ==1 ) +if (nr <=0 )
I was wondering if this would preserve the current behavior with SIS_FAST toggled off? Since the implementation below still does a "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?
Have you tried something similar to the below hunk?
/* because !--nr is the condition to stop scan */ nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1; if (nr == 1) return -1;
I agree with Mike that looking at slice to limit scan-depth seems odd. My experience with netperf is that the workload cares more about the server-client being co-located on the closest cache domain and by limiting scan-depth using slice, this is indirectly achieved since all the wakeups carry the WF_SYNc flag.
P.S. have you tried using the slice in __select_idle_cpu()? Similar to sched_idle_cpu() check, perhaps an additional sched_preempt_short_cpu() which compares rq->curr->se.slice with the waking task's slice and returs that cpu if SIS_SHORT can help run the workload quicker? Note:
This will not work if the SIS scan itself is the largest overhead in the wakeup cycle and not the task placement itself. Previously during SIS_UTIL testing, to measure the overheads of scan vs placement, we would do a full scan but return the result that SIS_UTIL would have returned to determine the overhead of the search itself.
Quoted 8 lines
kernel/sched/fair.c +0
-0return -1 ; } }kernel/sched/features.h +0
-0
kernel/sched/features.h SCHED_FEAT(TTWU_QUEUE, true) · line 56
Quoted 7 lines
kernel/sched/features.h +1
-0@@ -50,6 +50,7 @@ SCHED_FEAT(TTWU_QUEUE, true)* When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SIS_FAST , true)/* * Issue a WARN when we do multiple update_rq_clock() calls
--
Thanks and Regards,
Prateek
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Hi Prateek,
Quoted 120 lines · K Prateek Nayak
Hello Chenyu,
Quoted 104 lines · Chen Yu
Quoted 6 lines · Peter Zijlstra
Allow applications to directly set a suggested request/slice length using
sched_attr::sched_runtime.The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Quoted 7 lines
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Is it possible to leverage this task slice to do better task wakeup placement? The idea is that, the smaller the slice the wakee has, the less idle CPU it should scan. This can reduce wake latency and inhibit costly task migration, especially on large systems.
We did some experiments and got some performance improvements:
From
9cb806476586d7048fcbd0f66d0101f0dbb8fd2bMon Sep 17 00:00:00 2001
From: Chen Yu <[email protected]>
Date: Tue, 7 May 2024 22:36:29 +0800
Subject: [RFC PATCH] sched/eevdf: Use customized slice to reduce wakeup latency
and inhibit task migrationProblem 1:
The overhead of task migration is high on many-core system. The overhead brings performance penalty due to broken cache locality/higher cache-to-cache latency.Problem 2:
During wakeup, the time spent on searching for an idle CPU is costly on many-core system. Besides, access to other CPU's rq statistics brings cace contention:
available_idle_cpu(cpu)->idle_cpu(cpu)-> {rq->curr,rq->nr_running}Although
SIS_UTILthrottles the scan depth based on system utilization, there is requirement to further limit the scan depth for specific workload, especially for short duration wakee.Now we have the interface to customize the request/slice. The smaller the slice is, the earlier the task can be picked up, and the lower wakeup latency the task expects. Leverage the wakee's slice to further throttle the idle CPU scan depth - the shorter slice, the less CPUs to scan.
Test on 240 CPUs, 2 sockets system. With SNC(sub-numa-cluster) enabled, each LLC domain has 60 CPUs. There is noticeable improvement of netperf. (With SNC disabled, more improvements should be seen because C2C is higher)
The global slice is 3
msec(sysctl_sched_base_slice)by default on my ubuntu 22.04, and the customized slice is set to 0.1 msec for both netperf and netserver:for i in $(seq 1 $job); do
netperf_slice -e 100000 -4 -H 127.0.01 -t TCP_RR -c -C -l 100 &done
case load baseline(std%) compare%( std%) TCP_RR 60-threads 1.00 ( 1.60) +0.35 ( 1.73) TCP_RR 120-threads 1.00 ( 1.34) -0.96 ( 1.24) TCP_RR 180-threads 1.00 ( 1.59) +92.20 ( 4.24) TCP_RR 240-threads 1.00 ( 9.71) +43.11 ( 2.97)Suggested-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>--- kernel/sched/fair.c | 23 ++++++++++++++++++++--- kernel/sched/features.h | 1 + 2 files changed, 21 insertions(+), 3 deletions(-)kernel/sched/fair.c +18
-0@@ -7368,6 +7368,24 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +/* +* Scale the scan number of idle CPUs according to customized +* wakee's slice. The smaller the slice is, the earlier the task +* wants be picked up, thus the lower wakeup latency the task expects. +* The baseline is the global sysctl_sched_base_slice. Task slice +* smaller than the global one would shrink the scan number. +*/ +static int adjust_idle_scan (struct task_struct *p,int nr) +{ +if (!sched_feat (SIS_FAST )) +return nr; + +if (!p->se.custom_slice || p->se.slice >= sysctl_sched_base_slice) +return nr; + +return div_u64 (nr * p->se.slice, sysctl_sched_base_slice); +} +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7407
Quoted 8 lines
kernel/sched/fair.c +2
-2@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolif (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) { - /* because !--nr is the condition to stop scan */> - nr = READ_ONCE(sd_share->nr_idle_scan) + 1; + nr =adjust_idle_scan (p,READ_ONCE (sd_share->nr_idle_scan));/* overloaded LLC is unlikely to have idle cpu/core */ -if (nr ==1 ) +if (nr <=0 )I was wondering if this would preserve the current behavior with
SIS_FASTtoggled off? Since the implementation below still does a "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?
Quoted 7 lines
Have you tried something similar to the below hunk?
/* because !--nr is the condition to stop scan */ nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1; if (nr == 1) return -1;
Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
Quoted 6 lines
I agree with Mike that looking at slice to limit scan-depth seems odd. My experience with netperf is that the workload cares more about the server-client being co-located on the closest cache domain and by limiting scan-depth using slice, this is indirectly achieved since all the wakeups carry the
WF_SYNcflag.
Exactly. This is the original motivation.
Quoted 4 lines
P.S. have you tried using the slice in__select_idle_cpu()? Similar tosched_idle_cpu()check, perhaps an additionalsched_preempt_short_cpu()which comparesrq->curr->se.slicewith the waking task's slice and returs that cpu ifSIS_SHORTcan help run the workload quicker?
This is a good idea, it seems to be benefit PREEMPT_SHORT. If the customized task slice is introduced, we can leverage this hint for latency related optimization. Task wakeup is one thing, I can also think of other aspects, like idle load balance, etc. I'm not sure what is the proper usage of the task slice though, this is why I sent this RFC.
Quoted 7 lines
Note:
This will not work if the SIS scan itself is the largest overhead in the wakeup cycle and not the task placement itself. Previously duringSIS_UTILtesting, to measure the overheads of scan vs placement, we would do a full scan but return the result thatSIS_UTILwould have returned to determine the overhead of the search itself.
Regarding the task placement, do you mean the time between a task is enqueued and picked up? Do you have any recommendation which workload can expose the scan overhead most?
thanks,
Chenyu
Quoted 19 lines
Quoted 8 lines
kernel/sched/fair.c +0
-0return -1 ; } }kernel/sched/features.h +0
-0kernel/sched/features.h SCHED_FEAT(TTWU_QUEUE, true) · line 56
Quoted 7 lines
kernel/sched/features.h +1
-0@@ -50,6 +50,7 @@ SCHED_FEAT(TTWU_QUEUE, true)* When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SIS_FAST , true)/* * Issue a WARN when we do multiple update_rq_clock() calls --
Thanks and Regards,
Prateek
K Prateek Nayak
reviewerRe: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Hello Chenyu,
Quoted 56 lines · Chen Yu
Quoted 18 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7407
Quoted 13 lines
[..snip..]
/*
* Scan the LLC domain for idle CPUs; this is dynamically regulated by
* comparing the average scan cost (tracked insd->avg_scan_cost) against thekernel/sched/fair.c +2
-2@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolif (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) { - /* because !--nr is the condition to stop scan */> - nr = READ_ONCE(sd_share->nr_idle_scan) + 1; + nr =adjust_idle_scan (p,READ_ONCE (sd_share->nr_idle_scan));/* overloaded LLC is unlikely to have idle cpu/core */ -if (nr ==1 ) +if (nr <=0 )I was wondering if this would preserve the current behavior with
SIS_FASTtoggled off? Since the implementation below still does a "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?Quoted 7 lines
Have you tried something similar to the below hunk?
/* because !--nr is the condition to stop scan */ nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1; if (nr == 1) return -1;Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
Quoted 6 lines
I agree with Mike that looking at slice to limit scan-depth seems odd. My experience with netperf is that the workload cares more about the server-client being co-located on the closest cache domain and by limiting scan-depth using slice, this is indirectly achieved since all the wakeups carry the
WF_SYNcflag.Exactly. This is the original motivation.
Quoted 4 lines
P.S. have you tried using the slice in__select_idle_cpu()? Similar tosched_idle_cpu()check, perhaps an additionalsched_preempt_short_cpu()which comparesrq->curr->se.slicewith the waking task's slice and returs that cpu ifSIS_SHORTcan help run the workload quicker?This is a good idea, it seems to be benefit
PREEMPT_SHORT. If the customized task slice is introduced, we can leverage this hint for latency related optimization. Task wakeup is one thing, I can also think of other aspects, like idle load balance, etc. I'm not sure what is the proper usage of the task slice though, this is why I sent this RFC.Quoted 7 lines
Note:
This will not work if the SIS scan itself is the largest overhead in the wakeup cycle and not the task placement itself. Previously duringSIS_UTILtesting, to measure the overheads of scan vs placement, we would do a full scan but return the result thatSIS_UTILwould have returned to determine the overhead of the search itself.
Quoted 3 lines
Regarding the task placement, do you mean the time between a task is enqueued and picked up? Do you have any recommendation which workload can expose the scan overhead most?
Sorry for not being clear here. From what I've observed in the past, there are two dimensions to slect_idle_sibling():
i) Placement: Final CPU select_idle_sibling() returns ii) Search: Do we find an idle core/CPU in select_idle_sibling()
In case of netperf, I've observed that i) is more important than ii) wherin a placement of client on same core/thread as that of the server results in better performance vs finding an idle CPU on a remote LLC. For hackbench/tbench, when runqueues are under high utilization (~75%), reduction in search time ii) seems to be more beneficial.
There was also a wakeup from IPI / without IPI angle that I never quite got to the bottom of that Mathieu has highlighted last year. I'll go get some more data on that front and give your patch a try. Expect results in a couple of days.
Thank you.
Quoted 8 lines
thanks,
ChenyuQuoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +0
-0return -1 ; } }
--
Thanks and Regards,
Prateek
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Quoted 63 lines · K Prateek Nayak
Hello Chenyu,
Quoted 56 lines · Chen Yu
Quoted 18 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7407
Quoted 13 lines
[..snip..]
/*
* Scan the LLC domain for idle CPUs; this is dynamically regulated by
* comparing the average scan cost (tracked insd->avg_scan_cost) against thekernel/sched/fair.c +2
-2@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolif (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) { - /* because !--nr is the condition to stop scan */> - nr = READ_ONCE(sd_share->nr_idle_scan) + 1; + nr =adjust_idle_scan (p,READ_ONCE (sd_share->nr_idle_scan));/* overloaded LLC is unlikely to have idle cpu/core */ -if (nr ==1 ) +if (nr <=0 )I was wondering if this would preserve the current behavior with
SIS_FASTtoggled off? Since the implementation below still does a "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?Quoted 7 lines
Have you tried something similar to the below hunk?
/* because !--nr is the condition to stop scan */ nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1; if (nr == 1) return -1;Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
Quoted 6 lines
I agree with Mike that looking at slice to limit scan-depth seems odd. My experience with netperf is that the workload cares more about the server-client being co-located on the closest cache domain and by limiting scan-depth using slice, this is indirectly achieved since all the wakeups carry the
WF_SYNcflag.Exactly. This is the original motivation.
Quoted 4 lines
P.S. have you tried using the slice in__select_idle_cpu()? Similar tosched_idle_cpu()check, perhaps an additionalsched_preempt_short_cpu()which comparesrq->curr->se.slicewith the waking task's slice and returs that cpu ifSIS_SHORTcan help run the workload quicker?This is a good idea, it seems to be benefit
PREEMPT_SHORT. If the customized task slice is introduced, we can leverage this hint for latency related optimization. Task wakeup is one thing, I can also think of other aspects, like idle load balance, etc. I'm not sure what is the proper usage of the task slice though, this is why I sent this RFC.Quoted 7 lines
Note:
This will not work if the SIS scan itself is the largest overhead in the wakeup cycle and not the task placement itself. Previously duringSIS_UTILtesting, to measure the overheads of scan vs placement, we would do a full scan but return the result thatSIS_UTILwould have returned to determine the overhead of the search itself.Quoted 3 lines
Regarding the task placement, do you mean the time between a task is enqueued and picked up? Do you have any recommendation which workload can expose the scan overhead most?
Quoted 6 lines
Sorry for not being clear here. From what I've observed in the past, there are two dimensions to
slect_idle_sibling():i) Placement: Final CPU
select_idle_sibling()returns ii) Search: Do we find an idle core/CPU inselect_idle_sibling()
I see.
Quoted 3 lines
In case of netperf, I've observed that i) is more important than ii) wherin a placement of client on same core/thread as that of the server results in better performance vs finding an idle CPU on a remote LLC.
How about placement of client on same core/thread vs finding an idle CPU on a local LLC?
Quoted 3 lines
For hackbench/tbench, when runqueues are under high utilization (~75%), reduction in search time ii) seems to be more beneficial.
I can understand that hackbench is idle-cpu sensitive because it is MxN wakeup relationship and could result in task stacking easily. While for tbench, it should be similar to netperf, not sure why it does not fall into i) > ii) case. Is it because you were testing netperf TCP_RR(full-duplex) and tbench is half-duplex(the former has stronger cache locality requirement)?
Quoted 4 lines
There was also a wakeup from IPI / without IPI angle that I never quite got to the bottom of that Mathieu has highlighted last year. I'll go get some more data on that front and give your patch a try. Expect results in a couple of days.
Sure, I'd be glad to try your patch.
thanks,
Chenyu
Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to K Prateek NayakQuoted 66 lines · K Prateek Nayak
Hello Chenyu,
Quoted 56 lines · Chen Yu
Quoted 18 lines
Review static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7407
Quoted 13 lines
[..snip..]
/*
* Scan the LLC domain for idle CPUs; this is dynamically regulated by
* comparing the average scan cost (tracked insd->avg_scan_cost) against the(quoted diff) +2
-2@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool if (sched_feat(SIS_UTIL)) { sd_share = rcu_dereference(per_cpu(sd_llc_shared, target)); if (sd_share) { - /* because !--nr is the condition to stop scan */> - nr = READ_ONCE(sd_share->nr_idle_scan) + 1; + nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)); /* overloaded LLC is unlikely to have idle cpu/core */ - if (nr == 1) + if (nr <= 0)I was wondering if this would preserve the current behavior with
SIS_FASTtoggled off? Since the implementation below still does a "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?Quoted 7 lines
Have you tried something similar to the below hunk?
/* because !--nr is the condition to stop scan */ nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1; if (nr == 1) return -1;Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
Quoted 6 lines
I agree with Mike that looking at slice to limit scan-depth seems odd. My experience with netperf is that the workload cares more about the server-client being co-located on the closest cache domain and by limiting scan-depth using slice, this is indirectly achieved since all the wakeups carry the
WF_SYNcflag.Exactly. This is the original motivation.
Quoted 4 lines
P.S. have you tried using the slice in__select_idle_cpu()? Similar tosched_idle_cpu()check, perhaps an additionalsched_preempt_short_cpu()which comparesrq->curr->se.slicewith the waking task's slice and returs that cpu ifSIS_SHORTcan help run the workload quicker?This is a good idea, it seems to be benefit
PREEMPT_SHORT. If the customized task slice is introduced, we can leverage this hint for latency related optimization. Task wakeup is one thing, I can also think of other aspects, like idle load balance, etc. I'm not sure what is the proper usage of the task slice though, this is why I sent this RFC.Quoted 7 lines
Note:
This will not work if the SIS scan itself is the largest overhead in the wakeup cycle and not the task placement itself. Previously duringSIS_UTILtesting, to measure the overheads of scan vs placement, we would do a full scan but return the result thatSIS_UTILwould have returned to determine the overhead of the search itself.Quoted 3 lines
Regarding the task placement, do you mean the time between a task is enqueued and picked up? Do you have any recommendation which workload can expose the scan overhead most?
Sorry for not being clear here. From what I've observed in the past, there are two dimensions to
slect_idle_sibling():
Quoted 9 lines
i) Placement: Final CPU
select_idle_sibling()returns ii) Search: Do we find an idle core/CPU inselect_idle_sibling()In case of netperf, I've observed that i) is more important than ii) wherin a placement of client on same core/thread as that of the server results in better performance vs finding an idle CPU on a remote LLC. For hackbench/tbench, when runqueues are under high utilization (~75%), reduction in search time ii) seems to be more beneficial.
Usually task stacking is not preferred because it hurts latency. But as we discovered on AMD and Intel's large servers, sometimes the overhead of task migration offset the benefit of running an idle CPU.
Inspired by your description on SIS_SHORT, and also Mike's feedback, I respin the SIS_SHORT patch, to consider the task's customized slice:
Leverage the WF_SYNC to achieve better cache locality.
If both the waker and the wakee are short duration tasks, wake up the wakee on waker's CPU directly. To avoid task stacking as much as possible:
1. Only when has_idle_core is false, SIS_SYNC takes effect.
2. Only if the user has shrinked the task slice, SIS_SYNC takes effect.
3. The waker must be the only runnable task on the runqueue.
4. The duration threshold is based on the CPU number within the LLC. The more CPUs there are, the lower the bar for the task to be treated as a short duration one, and vice versa.
threshold = sysctl_sched_migration_cost * llc_weight^2 / 256^2
threshold
LLC_WEIGHT=8 0.5 usec LLC_WEIGHT=16 2 usec LLC_WEIGHT=32 8 usec LLC_WEIGHT=64 31 usec LLC_WEIGHT=128 125 usec LLC_WEIGHT=256 500 usec
5. Honor idle-CPU-first for CPUs share the L2 cache(Cluster). Only there is no idle CPU within the Cluster domain, SIS_SYNC takes effect.
Benchmark:
Tested on 4 platforms, significant throughput improvement on tbench, netperf, stress-ng, will-it-scale, and latency reduced of lmbench. No performance difference was observed in an Online Transaction Processing (OLTP) test.
Platform1, 240 CPUs, 2 sockets Intel(R) Xeon(R)
========================================================================
netperf
=======
case load baseline(std%) compare%( std%) TCP_RR 60-threads 1.00 ( 1.11) +8.22 ( 1.06) TCP_RR 120-threads 1.00 ( 1.74) +38.71 ( 0.74) TCP_RR 180-threads 1.00 ( 1.21) +108.12 ( 1.07) TCP_RR 240-threads 1.00 ( 4.75) +160.98 ( 4.01) UDP_RR 60-threads 1.00 ( 11.70) +7.17 ( 10.89) UDP_RR 120-threads 1.00 ( 10.82) +5.54 ( 7.17) UDP_RR 180-threads 1.00 ( 9.77) +10.29 ( 10.87) UDP_RR 240-threads 1.00 ( 12.38) +4.45 ( 16.69)
tbench
======
case load baseline(std%) compare%( std%) loopback 60-threads 1.00 ( 0.22) +8.67 ( 0.12) loopback 120-threads 1.00 ( 0.20) +42.12 ( 1.16) loopback 180-threads 1.00 ( 0.16) +96.36 ( 1.34) loopback 240-threads 1.00 ( 0.23) +122.68 ( 4.87)
schbench
========
No noticeable difference of 99.0th wakeup/request latency, 50.0th RPS percentiles. schbench -m 2 -r 100
baseline sis_sync
Wakeup Latencies 99.0th usec 27 25
Request Latencies 99.0th usec 15376 15376 RPS percentiles 50.0th 16608 16608
Platform2, 48 CPUs 2 sockets Intel(R) Xeon(R) CPU E5-2697
========================================================================
lmbench3: lmbench3.PIPE.latency.us 33.8% improvement
lmbench3: lmbench3.AF_UNIX.sock.stream.latency.us 30.6% improvement
Platform3: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480
=======================================================================
stress-ng: stress-ng.vm-rw.ops_per_sec 250.8% improvement
will-it-scale: will-it-scale.per_process_ops 42.1% improvement
Platform4: 288 CPUs, 2 sockets, 4 CPUs share the L2 cache, Intel(R) Xeon(R)
=========================================================================
netperf
=======
case load baseline(std%) compare%( std%) TCP_RR 72-threads 1.00 ( 0.89) +6.14 ( 0.95) TCP_RR 144-threads 1.00 ( 1.53) +5.18 ( 1.24) TCP_RR 216-threads 1.00 ( 10.76) +23.31 ( 1.67) TCP_RR 288-threads 1.00 ( 2.28) +1.72 ( 2.10) UDP_RR 72-threads 1.00 ( 3.44) +0.42 ( 3.11) UDP_RR 144-threads 1.00 ( 14.53) +1.40 ( 16.11) UDP_RR 216-threads 1.00 ( 17.30) +1.18 ( 23.00) UDP_RR 288-threads 1.00 ( 21.30) -0.15 ( 20.14)
kernel/sched/fair.c +39 -5
@@ -1003,7 +1003,7 @@ static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)#include "pelt.h" #ifdef CONFIG_SMP -static int select_idle_sibling (struct task_struct *p,int prev_cpu,int cpu); +static int select_idle_sibling (struct task_struct *p,int prev_cpu,int cpu,int sync);static unsigned long task_h_load (struct task_struct *p);static unsigned long capacity_of (int cpu);
@@ -7410,16 +7410,36 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +static int short_task (struct task_struct *p,int llc) +{ +if (!p->se.custom_slice || p->se.slice >= sysctl_sched_base_slice) +return 0 ; + +/* +* Scale the threshold by the LLC CPU number. +* The more CPUs there are, the more likely the +* task is regarded as a small one. +* +*/ +if ((p->duration_avg <<16 ) >= + (sysctl_sched_migration_cost * llc * llc)) +return 0 ; + +return 1 ; +} +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the * average idle time for this rq (as found in rq->avg_idle). */ -static int select_idle_cpu (struct task_struct *p,struct sched_domain *sd,bool has_idle_core,int target) +static int select_idle_cpu (struct task_struct *p,struct sched_domain *sd,bool has_idle_core,int target, +int sync) {struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_rq_mask);int i, cpu, idle_cpu =-1 , nr =INT_MAX ;struct sched_domain_shared *sd_share; +int llc_weight =per_cpu (sd_llc_size, target);cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr);
@@ -7458,6 +7478,20 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool } } +/* +* When reached here, next step is to scan idle CPUs that do not share L2. +* However, the Core-to-Core cache latency could be large on big system. +* Give target another chance if waker and wakee are mutually waking up +* each other. +*/ +if (sched_feat (SIS_SYNC ) && + target ==smp_processor_id () && !has_idle_core && + sync &&this_rq ()->nr_running <=1 && +short_task (p, llc_weight) && +short_task (current, llc_weight)) { +return target; + } +for_each_cpu_wrap (cpu, cpus, target +1 ) {if (has_idle_core) { i =select_idle_core (p, cpu, cpus, &idle_cpu);
@@ -7550,7 +7584,7 @@ static inline bool asym_fits_cpu(unsigned long util,/* * Try and locate an idle core/thread in the LLC cache domain. */ -static int select_idle_sibling (struct task_struct *p,int prev,int target) +static int select_idle_sibling (struct task_struct *p,int prev,int target,int sync) {bool has_idle_core = false;struct sched_domain *sd;
@@ -7659,7 +7693,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) } } - i =select_idle_cpu (p, sd, has_idle_core, target); + i =select_idle_cpu (p, sd, has_idle_core, target, sync);if ((unsigned )i < nr_cpumask_bits)return i;
@@ -8259,7 +8293,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) new_cpu =sched_balance_find_dst_cpu (sd, p, cpu, prev_cpu, sd_flag); }else if (wake_flags &WF_TTWU ) {/* XXX always ? */ /* Fast path */ - new_cpu =select_idle_sibling (p, prev_cpu, new_cpu); + new_cpu =select_idle_sibling (p, prev_cpu, new_cpu, sync); }rcu_read_unlock ();
kernel/sched/features.h +1 -0
@@ -50,6 +50,7 @@ SCHED_FEAT(TTWU_QUEUE, true)* When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SIS_SYNC , true)/* * Issue a WARN when we do multiple update_rq_clock() calls
Signature · 5 lines
2.25.1 Preparation patch to record the task's duration:
include/linux/sched.h +3 -0
@@ -1339,6 +1339,9 @@ struct task_struct {struct callback_head cid_work;#endif +u64 prev_sleep_sum_runtime; +u64 duration_avg; +struct tlbflush_unmap_batch tlb_ubc;/* Cache last used pipe for splice(): */
kernel/sched/core.c +2 -0
@@ -4566,6 +4566,8 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending =NULL ;#endif init_sched_mm_cid (p); + p->prev_sleep_sum_runtime =0 ; + p->duration_avg =0 ; }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);
kernel/sched/fair.c +12 -0
@@ -6833,6 +6833,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)static void set_next_buddy (struct sched_entity *se); +static inline void dur_avg_update (struct task_struct *p) +{ +u64 dur; + + dur = p->se.sum_exec_runtime - p->prev_sleep_sum_runtime; + p->prev_sleep_sum_runtime = p->se.sum_exec_runtime; +update_avg (&p->duration_avg, dur); +} +/* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and
@@ -6905,6 +6914,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) dequeue_throttle:util_est_update (&rq->cfs, p, task_sleep); +if (task_sleep) +dur_avg_update (p); +hrtick_update (rq); }
Signature · 1 line
2.25.1
K Prateek Nayak
reviewerHello Peter,
Quoted 31 lines · Peter Zijlstra
Hi all,
I'm slowly crawling back out of the hole and trying to get back to work. Availability is still low on my end, but I'll try and respond to some email.
Anyway, in order to get my feet wet again with sitting behind a computer, find here a few patches that should functionally complete the EEVDF journey.
This very much includes the new interface that exposes the extra parameter that EEVDF has. I've chosen to use
sched_attr::sched_runtimefor this over a nice-like value because some workloads actually know their slice length (can be dynamically measured in the same way as for deadline usingCLOCK_THREAD_CPUTIME_ID) and using the real request size is much more effective than some relative measure.[[ using too short a request size will increase job preemption overhead,
using too long a request size will decrease timeliness ]]The whole delayed-dequeue thing is I think a fundamental thing that was missing from the EEVDF paper. Without something like this EEVDF will simply not work right. IIRC this was mentioned to me many years ago when people worked on BFQ iosched and ran into this same issue. Time had erased the critical aspect of this note and I had to re-discover it again.
Also, I think Ben expressed concern that preserving lag over long periods doesn't make sense a while back.
The implementation presented here is one that should work with our cgroup mess and keeps most of the ugly inside
fair.cunlike previous versions which puked all over the core scheduler code.
Quoted 8 lines
Critically cfs-cgroup throttling is not tested, and cgroups are only tested in so far that a systemd infected machine now boots (took a bit).
Other than that, it works well enough to build the next kernel and it passes the few trivial latency-slice tests I ran.
Anyway, please have a poke and let me know...
Sorry for the delay, I was waiting for all the basic issues to get fixed before giving it a test but since OSPM is this week and folks would want to discuss the series, I'm sharing the results based on the current
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/eevdf
At the start of the testing the HEAD was at commit 3883d848e41a ("sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion") and I'm using commit 4c43611fe406 ("Merge branch 'tip/sched/urgent'") as the base for comparison.
tl;dr
Mixed bag of results. We have hackbench and netperf slightly unhappy. stream is as is, no surprises there, and tbench and schbench seems to have a swing from being unhappy to happy as we scale. I'll leave full results below including some experiments around sched_feat(RESPECT_SLICE) proposed here
https://lore.kernel.org/lkml/[email protected]/
o System Details
- 3rd Generation EPYC System
- 2 x 64C/128T
- NPS1 mode
o Kernels
incomplete: peterz/queue:sched/eevdf at commit 4c43611fe406
("Merge branch 'tip/sched/urgent'")
complete: peterz/queue:sched/eevdf at commit 3883d848e41a
("sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion")
Note: The above mentioned tree is prove to force updates and the commit IDs reflect the state from when the branch was last updated on "2024-04-30"
o Results
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
Case: incomplete[pct imp](CV) complete[pct imp](CV) 1-groups 1.00 [ -0.00]( 3.42) 1.32 [-32.19]( 2.87) 2-groups 1.00 [ -0.00]( 1.28) 1.39 [-39.10]( 1.70) 4-groups 1.00 [ -0.00]( 0.93) 1.50 [-49.85]( 1.28) 8-groups 1.00 [ -0.00]( 1.00) 1.67 [-66.81]( 1.39) 16-groups 1.00 [ -0.00]( 2.69) 1.68 [-68.05]( 1.79)
==================================================================
Test : tbench
Units : Normalized throughput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: incomplete[pct imp](CV) complete[pct imp](CV)
1 1.00 [ 0.00]( 0.41) 0.98 [ -2.50]( 0.38)
2 1.00 [ 0.00]( 0.48) 0.96 [ -4.17]( 3.05)
4 1.00 [ 0.00]( 0.43) 0.96 [ -4.18]( 1.45)
8 1.00 [ 0.00]( 0.97) 0.93 [ -6.77]( 0.97)
16 1.00 [ 0.00]( 0.21) 0.91 [ -8.88]( 1.74)
32 1.00 [ 0.00]( 1.11) 0.92 [ -8.31]( 2.67)
64 1.00 [ 0.00]( 1.81) 0.96 [ -3.80]( 2.41)
128 1.00 [ 0.00]( 0.44) 1.02 [ 2.43]( 0.92)
256 1.00 [ 0.00]( 0.26) 0.96 [ -4.35]( 1.07)
512 1.00 [ 0.00]( 0.12) 0.97 [ -2.87]( 0.64)
1024 1.00 [ 0.00]( 0.32) 0.97 [ -2.59]( 0.26)==================================================================
Test : stream-10
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: incomplete[pct imp](CV) complete[pct imp](CV) Copy 1.00 [ 0.00](11.28) 0.80 [-20.19](15.70) Scale 1.00 [ 0.00]( 5.61) 0.99 [ -1.27]( 6.81) Add 1.00 [ 0.00]( 6.76) 0.91 [ -8.97]( 7.97) Triad 1.00 [ 0.00]( 4.76) 0.88 [-12.27]( 8.45)
==================================================================
Test : stream-100
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: incomplete[pct imp](CV) complete[pct imp](CV) Copy 1.00 [ 0.00]( 3.28) 0.95 [ -4.61]( 3.81) Scale 1.00 [ 0.00]( 2.00) 0.99 [ -0.87]( 5.04) Add 1.00 [ 0.00]( 1.44) 0.99 [ -0.51]( 2.12) Triad 1.00 [ 0.00]( 2.41) 1.01 [ 0.63]( 1.45)
==================================================================
Test : netperf
Units : Normalized Througput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: incomplete[pct imp](CV) complete[pct imp](CV) 1-clients 1.00 [ 0.00]( 0.33) 0.99 [ -0.85]( 1.65) 2-clients 1.00 [ 0.00]( 0.58) 0.99 [ -1.18]( 0.35) 4-clients 1.00 [ 0.00]( 0.32) 0.99 [ -0.78]( 1.27) 8-clients 1.00 [ 0.00]( 0.31) 0.99 [ -1.31]( 0.97) 16-clients 1.00 [ 0.00]( 0.53) 0.98 [ -1.95]( 1.11) 32-clients 1.00 [ 0.00]( 0.50) 0.97 [ -2.94]( 1.29) 64-clients 1.00 [ 0.00]( 1.71) 0.95 [ -4.86]( 1.14) 128-clients 1.00 [ 0.00]( 0.82) 0.99 [ -0.67]( 2.55) 256-clients 1.00 [ 0.00]( 3.88) 0.92 [ -7.65]( 3.27) 512-clients 1.00 [ 0.00](43.49) 0.94 [ -6.35](49.81)
==================================================================
Test : schbench (Old)
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: incomplete[pct imp](CV) complete[pct imp](CV) 1 1.00 [ -0.00](34.42) 0.76 [ 23.68](31.33) 2 1.00 [ -0.00]( 6.45) 0.83 [ 16.67]( 5.84) 4 1.00 [ -0.00](14.87) 0.77 [ 22.73](16.44) 8 1.00 [ -0.00](11.30) 1.07 [ -6.52]( 9.55) 16 1.00 [ -0.00]( 6.33) 1.03 [ -3.45]( 6.11) 32 1.00 [ -0.00]( 2.18) 0.94 [ 6.25]( 3.77) 64 1.00 [ -0.00]( 7.23) 0.99 [ 0.51]( 5.93) 128 1.00 [ -0.00]( 6.76) 0.95 [ 5.00]( 5.17) 256 1.00 [ -0.00](11.05) 1.22 [-22.33]( 9.70) 512 1.00 [ -0.00]( 2.15) 0.77 [ 23.20]( 2.81)
--
o Regression Breakdown
Following are some data I could gather for the regression observed:
- Hackbench
The runtime of hackbench with EEVDF Complete is 24.81% more that that on the current base kernel. Following is the IBS profile comparing EEVDF Complete on the left to current base on the right:
Command: sudo perf record -a -e ibs_op// -- perf bench sched messaging -p -t -l 100000 -g 1
Overhead Command Shared Object Symbol Overhead Command Shared Object Symbol 4.42% sched-messaging [kernel.kallsyms] [k] osq_lock 6.30% sched-messaging [kernel.kallsyms] [k] osq_lock 3.76% sched-messaging [kernel.kallsyms] [k] _copy_from_iter 4.06% sched-messaging [kernel.kallsyms] [k] _copy_to_iter 3.67% sched-messaging [kernel.kallsyms] [k] srso_alias_safe_ret 3.98% sched-messaging [kernel.kallsyms] [k] _copy_from_iter 3.18% sched-messaging [kernel.kallsyms] [k] _copy_to_iter 3.85% sched-messaging [kernel.kallsyms] [k] mutex_spin_on_owner 2.80% sched-messaging [kernel.kallsyms] [k] pipe_read 3.50% sched-messaging [kernel.kallsyms] [k] srso_alias_safe_ret 2.72% sched-messaging [kernel.kallsyms] [k] mutex_spin_on_owner 3.33% sched-messaging [kernel.kallsyms] [k] inode_needs_update_time 2.69% sched-messaging [kernel.kallsyms] [k] pipe_write 2.39% sched-messaging [kernel.kallsyms] [k] pipe_read 2.49% sched-messaging [kernel.kallsyms] [k] syscall_exit_to_user_mode 2.36% sched-messaging [kernel.kallsyms] [k] syscall_exit_to_user_mode 2.40% swapper [kernel.kallsyms] [k] poll_idle <---- 2.30% sched-messaging [kernel.kallsyms] [k] pipe_write 2.31% sched-messaging [kernel.kallsyms] [k] inode_needs_update_time 2.23% sched-messaging [kernel.kallsyms] [k] mutex_lock 2.14% sched-messaging [kernel.kallsyms] [k] srso_alias_return_thunk 2.07% sched-messaging [kernel.kallsyms] [k] update_sd_lb_stats.constprop.0 2.13% sched-messaging [kernel.kallsyms] [k] aa_file_perm 2.02% sched-messaging [kernel.kallsyms] [k] ksys_write 1.99% sched-messaging [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.00% sched-messaging [kernel.kallsyms] [k] aa_file_perm 1.99% sched-messaging [kernel.kallsyms] [k] do_syscall_64 1.98% sched-messaging libc.so.6 [.] __GI___libc_write 1.92% sched-messaging [kernel.kallsyms] [k] ksys_write 1.94% sched-messaging [kernel.kallsyms] [k] do_syscall_64 1.90% sched-messaging [kernel.kallsyms] [k] __fdget_pos 1.92% sched-messaging [kernel.kallsyms] [k] srso_alias_return_thunk 1.89% sched-messaging [kernel.kallsyms] [k] mutex_lock 1.76% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64 1.88% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64 1.62% sched-messaging [kernel.kallsyms] [k] __mutex_lock.constprop.0 1.62% sched-messaging libc.so.6 [.] __GI___libc_write 1.48% sched-messaging [kernel.kallsyms] [k] __fdget_pos 1.61% sched-messaging [kernel.kallsyms] [k] vfs_read 1.47% sched-messaging [kernel.kallsyms] [k] vfs_read 1.54% sched-messaging [kernel.kallsyms] [k] apparmor_file_permission 1.31% sched-messaging [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.43% sched-messaging [kernel.kallsyms] [k] rep_movs_alternative 1.29% sched-messaging [kernel.kallsyms] [k] apparmor_file_permission 1.40% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe 1.25% sched-messaging libc.so.6 [.] read 1.36% sched-messaging [kernel.kallsyms] [k] ksys_read 1.22% sched-messaging [kernel.kallsyms] [k] mutex_unlock 1.35% sched-messaging [kernel.kallsyms] [k] vfs_write 1.18% sched-messaging [kernel.kallsyms] [k] rep_movs_alternative 1.24% sched-messaging libc.so.6 [.] read 1.17% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe 1.23% sched-messaging [kernel.kallsyms] [k] srso_alias_untrain_ret 1.11% sched-messaging [kernel.kallsyms] [k] atime_needs_update 1.15% sched-messaging [kernel.kallsyms] [k] __mutex_lock.constprop.0 1.10% sched-messaging [kernel.kallsyms] [k] vfs_write 1.15% sched-messaging [kernel.kallsyms] [k] atime_needs_update 1.08% sched-messaging [kernel.kallsyms] [k] ksys_read 1.11% swapper [kernel.kallsyms] [k] psi_group_change 1.03% sched-messaging [kernel.kallsyms] [k] srso_alias_untrain_ret 1.03% sched-messaging [kernel.kallsyms] [k] copy_page_from_iter 1.01% swapper [kernel.kallsyms] [k] psi_group_change 0.98% sched-messaging [kernel.kallsyms] [k] update_sd_lb_stats.constprop.0 0.96% sched-messaging [kernel.kallsyms] [k] copy_page_from_iter 0.95% sched-messaging [kernel.kallsyms] [k] psi_group_change 0.93% sched-messaging [kernel.kallsyms] [k] psi_group_change 0.91% sched-messaging [kernel.kallsyms] [k] copy_page_to_iter 0.79% sched-messaging [kernel.kallsyms] [k] copy_page_to_iter 0.89% sched-messaging [kernel.kallsyms] [k] rw_verify_area 0.76% swapper [kernel.kallsyms] [k] poll_idle <------ poll idle jumps up in case of 0.86% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter 0.71% sched-messaging [kernel.kallsyms] [k] rw_verify_area EEVDF complete 0.77% sched-messaging [kernel.kallsyms] [k] mutex_unlock 0.71% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter 0.77% sched-messaging [kernel.kallsyms] [k] security_file_permission 0.69% sched-messaging [kernel.kallsyms] [k] try_to_wake_up 0.73% sched-messaging [kernel.kallsyms] [k] select_task_rq_fair 0.67% sched-messaging [kernel.kallsyms] [k] select_task_rq_fair 0.71% sched-messaging [kernel.kallsyms] [k] __schedule 0.63% sched-messaging [kernel.kallsyms] [k] __schedule 0.70% sched-messaging [kernel.kallsyms] [k] entry_SYSRETQ_unsafe_stack 0.61% sched-messaging [kernel.kallsyms] [k] entry_SYSRETQ_unsafe_stack 0.59% sched-messaging [kernel.kallsyms] [k] syscall_return_via_sysret 0.61% sched-messaging [kernel.kallsyms] [k] security_file_permission 0.57% sched-messaging [kernel.kallsyms] [k] dequeue_entity 0.52% sched-messaging [kernel.kallsyms] [k] update_load_avg 0.54% sched-messaging [kernel.kallsyms] [k] current_time 0.51% sched-messaging [kernel.kallsyms] [k] native_sched_clock
--
Further pinning the workload to one LLC domain (CCX) shows poll_idle jump to top:
Command: sudo perf record -C 0-7,128-131 -e ibs_op// -- taskset -c 0-7,128-131 perf bench sched messaging -p -t -l 100000 -g 1
Overhead Command Shared Object Symbol Overhead Command Shared Object Symbol 9.14% swapper [kernel.kallsyms] [k] poll_idle <----- 3.55% sched-messaging [kernel.kallsyms] [k] pipe_read 3.02% sched-messaging [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.53% sched-messaging [kernel.kallsyms] [k] pipe_write 3.00% sched-messaging [kernel.kallsyms] [k] pipe_read 3.48% sched-messaging [kernel.kallsyms] [k] srso_alias_safe_ret 2.97% sched-messaging [kernel.kallsyms] [k] pipe_write 3.37% sched-messaging [kernel.kallsyms] [k] _copy_from_iter 2.95% sched-messaging [kernel.kallsyms] [k] srso_alias_safe_ret 3.15% sched-messaging [kernel.kallsyms] [k] syscall_exit_to_user_mode 2.83% sched-messaging [kernel.kallsyms] [k] _copy_from_iter 2.76% sched-messaging [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.56% sched-messaging [kernel.kallsyms] [k] syscall_exit_to_user_mode 2.62% sched-messaging [kernel.kallsyms] [k] srso_alias_return_thunk 2.23% sched-messaging [kernel.kallsyms] [k] srso_alias_return_thunk 2.59% sched-messaging [kernel.kallsyms] [k] do_syscall_64 2.21% sched-messaging [kernel.kallsyms] [k] do_syscall_64 2.44% sched-messaging [kernel.kallsyms] [k] __fdget_pos 2.05% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64 2.34% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64 2.04% sched-messaging [kernel.kallsyms] [k] __fdget_pos 2.20% sched-messaging [kernel.kallsyms] [k] _copy_to_iter 1.87% sched-messaging [kernel.kallsyms] [k] _copy_to_iter 2.18% sched-messaging [kernel.kallsyms] [k] aa_file_perm 1.82% sched-messaging [kernel.kallsyms] [k] aa_file_perm 2.16% sched-messaging [kernel.kallsyms] [k] vfs_read 1.67% sched-messaging [kernel.kallsyms] [k] apparmor_file_permission 2.08% swapper [kernel.kallsyms] [k] poll_idle <----- 1.65% sched-messaging [kernel.kallsyms] [k] vfs_read 2.03% sched-messaging [kernel.kallsyms] [k] apparmor_file_permission 1.63% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe 2.00% sched-messaging [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe 1.59% swapper [kernel.kallsyms] [k] psi_group_change 1.85% sched-messaging [kernel.kallsyms] [k] rep_movs_alternative 1.56% sched-messaging [kernel.kallsyms] [k] ksys_write 1.83% sched-messaging [kernel.kallsyms] [k] vfs_write 1.54% sched-messaging [kernel.kallsyms] [k] ksys_read 1.82% sched-messaging [kernel.kallsyms] [k] ksys_write 1.51% sched-messaging [kernel.kallsyms] [k] vfs_write 1.72% sched-messaging [kernel.kallsyms] [k] srso_alias_untrain_ret 1.49% sched-messaging [kernel.kallsyms] [k] rep_movs_alternative 1.71% sched-messaging libc.so.6 [.] __GI___libc_write 1.46% sched-messaging libc.so.6 [.] __GI___libc_write 1.70% sched-messaging [kernel.kallsyms] [k] ksys_read 1.42% sched-messaging [kernel.kallsyms] [k] srso_alias_untrain_ret 1.58% sched-messaging [kernel.kallsyms] [k] atime_needs_update 1.30% sched-messaging libc.so.6 [.] read 1.56% sched-messaging libc.so.6 [.] read 1.24% sched-messaging [kernel.kallsyms] [k] atime_needs_update 1.56% sched-messaging [kernel.kallsyms] [k] mutex_lock 1.20% sched-messaging [kernel.kallsyms] [k] mutex_lock 1.35% sched-messaging [kernel.kallsyms] [k] psi_group_change 1.04% sched-messaging [kernel.kallsyms] [k] psi_group_change 1.26% sched-messaging [kernel.kallsyms] [k] copy_page_to_iter 1.03% sched-messaging [kernel.kallsyms] [k] inode_needs_update_time 1.20% sched-messaging [kernel.kallsyms] [k] select_task_rq_fair 0.99% sched-messaging [kernel.kallsyms] [k] rw_verify_area 1.16% sched-messaging [kernel.kallsyms] [k] inode_needs_update_time 0.98% sched-messaging [kernel.kallsyms] [k] copy_page_to_iter 1.16% sched-messaging [kernel.kallsyms] [k] rw_verify_area 0.95% sched-messaging [kernel.kallsyms] [k] select_task_rq_fair 1.12% sched-messaging [kernel.kallsyms] [k] copy_page_from_iter 0.95% sched-messaging [kernel.kallsyms] [k] copy_page_from_iter 1.06% swapper [kernel.kallsyms] [k] psi_group_change 0.92% sched-messaging [kernel.kallsyms] [k] __schedule 1.04% sched-messaging [kernel.kallsyms] [k] security_file_permission 0.85% sched-messaging [kernel.kallsyms] [k] security_file_permission 0.96% sched-messaging [kernel.kallsyms] [k] try_to_wake_up 0.68% swapper [kernel.kallsyms] [k] do_idle 0.86% sched-messaging [kernel.kallsyms] [k] entry_SYSRETQ_unsafe_stack 0.67% sched-messaging [kernel.kallsyms] [k] dequeue_entity 0.80% sched-messaging [kernel.kallsyms] [k] mutex_spin_on_owner 0.67% sched-messaging [kernel.kallsyms] [k] entry_SYSRETQ_unsafe_stack 0.79% sched-messaging [kernel.kallsyms] [k] __schedule 0.66% sched-messaging [kernel.kallsyms] [k] update_load_avg 0.79% sched-messaging [kernel.kallsyms] [k] syscall_return_via_sysret 0.65% sched-messaging [kernel.kallsyms] [k] syscall_return_via_sysret 0.75% sched-messaging [kernel.kallsyms] [k] current_time 0.62% sched-messaging [kernel.kallsyms] [k] mutex_spin_on_owner 0.66% sched-messaging [kernel.kallsyms] [k] __mutex_lock.constprop.0 0.60% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter 0.63% sched-messaging perf [.] sender 0.60% swapper [kernel.kallsyms] [k] __schedule 0.63% sched-messaging [kernel.kallsyms] [k] update_load_avg 0.59% sched-messaging [kernel.kallsyms] [k] current_time 0.59% sched-messaging perf [.] receiver 0.54% swapper [kernel.kallsyms] [k] enqueue_entity 0.56% sched-messaging [kernel.kallsyms] [k] file_update_time 0.52% swapper [kernel.kallsyms] [k] menu_select 0.55% sched-messaging [kernel.kallsyms] [k] touch_atime 0.50% sched-messaging perf [.] sender 0.52% sched-messaging [kernel.kallsyms] [k] x64_sys_call
--
This jump up of poll_idle seems to be a higher order behavior change as a result of Complete EEVDF. Looking at "sched/fair: Implement delayed dequeue", can there be a case where ttwu_runnable() in try_to_wake_up() can stop task migration on wakeup and queue it on a busy queue where it was marked for a dealyed dequeue?
When looking at per-task statistics from sched-scoreboard, I see a 41% decrease in nr_migrations going from base to Complete EEVDF. I also see the cumulative wait sum shoot up by 88%.
comm: sched-messaging incomplete complete %diff
Average of avg_atom 0.118341601 0.036585796 -69%
Average of avg_per_cpu 0.446550378 0.668355323 50%
Sum of nr_migrations 1414593 829870 -41% *
Sum of nr_switches 4013420 3563675 -11%
Sum of nr_voluntary_switches 3912865 3562547 -9%
Sum of nr_involuntary_switches 100555 1128 -99%
Sum of nr_wakeups 3913621 3567097 -9%
Sum of sum_exec_runtime 95557.37505 101666.7651 6% Sum of sum_idle_runtime 1928546.553 975487.7551 -49% Sum of wait_sum 69884.24284 131511.9207 88% *
For tbench and netperf I could not get any concrete culprit from perf with IBS or per-task stats. I'll continue digging. However, it could just be similar higher order effect where poll_idle steals cycles from smt sibling running the task.
---------------------------------------------------------
- Experiments with sched_attr::sched_runtime on SPECjbb -
---------------------------------------------------------
I tried some experiments with sched_attr:sched_runtime on SPECjbb. With the base kernel, I was not able see any changes in the SPECjbb results with various sched_attr::sched_runtime set for various threads. However, with Complete EEVDF (contains RESPECT_SLICE), I'm able to eek out some performance improvements on the workload.
Background information on the threads of SPECjbb - txinject drives the workload by injecting transaction. MapReduce, ForkJoinPool, and the Backend threads are responsible for the main business logic of transaction processing. There are GC threads from Java and a compiler thread for what I presume is some form of JIT compilation.
I've used following naive strategy:
- Set a very small slice to txinject since I would like to push as much load as possible, as quickly as possible.
- Very small slice to ForkJoinPool and MapReduce threads so they can go ahead and pass the load to backend threads.
- Very small slice to compiler threads so thay can unblock whatever is waiting on the JIT compilation.
- A very large slice to the backend thread so it can process as much data as possible in one go.
- Maximum slice size to the GC thread purely because it should not interrupt other processes.
With the above setting, I see the following trend:
EEVDF Complete vs base
Max-jOPS +7.05%
Critical-jOPS -15.62%
The key takeaway of the experiment is that, with EEVDF complete, I'm actually starting to see some behavior difference with different values unlike with the current tip:Sched/core. I believe it is RESPECT_SLICE helping to keep tasks with large slices on the runqueue beyond the 0-lag point.
Note: These results are from naive experiments. I'm sure looking at the per-task statistics and deciding on the slices more judiciously can further benefit the workload.
On a parting note, it was not very straightforward to set these values outside of the application without any application modification. In my case, I was consuming "task_rename" evvents via trace pipe to find the relevant pids to then tag.
------------------------------------
- DeathStarBench and RESPECT_SLICE -
------------------------------------
Following is the trend I see for RESPECT_SLICE on DeathStarBench:
Pinning #instances EEVDF_COMPLETE
1LLC 1 9%
2LLC 2 3%
4LLC 3 -1%
8LLC 6 -5%
Dropping RESPECT_SLICE and using my original RUN_TO_PARITY_WAKEUP
https://lore.kernel.org/lkml/[email protected]/
still shows regression for larger instances pinned to larger number of LLC domains with EEVDF Complete. This remains to be investigated as well.
I'll update the thread with more information in the coming days. --
Thanks and Regards,
Prateek