Skip to contentSkip to conversation

sched_pair_cpu: Introduce scheduler task pairing system call

RFC PATCH In review Mathieu Desnoyers 4 participants 13 messages Jun 19, 2020 – Jun 26, 2020

Subsystems (8)
x86 VFS MM core Scheduler +4 more subsystems
lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/1 sched_pair_cpu: Introduce scheduler task pairing system call

There is an important use-case which is not possible with the "rseq" (Restartable Sequences) system call, which was left as future work.

That use-case is to modify user-space per-cpu data structures belonging to specific CPUs which may be brought offline and online again by CPU hotplug. This can be used by memory allocators to migrate free memory pools when CPUs are brought offline, or by ring buffer consumers to target specific per-CPU buffers, even when CPUs are brought offline.

A few rather complex prior attempts were made to solve this. Those were based on in-kernel interpreters (cpu_opv, do_on_cpu). That complexity was generally frowned upon, even by their author.

This patch fulfills this use-case in a simpler way: it introduces a "sched_pair_cpu" system call, which allows user-space threads to ask to run "as if" it was running on the runqueue of a specific CPU, but without being migrated to that cpu, until that pairing is cleared.

"But this can already be done with sched_setaffinity", some could be tempted to reply. However, sched_setaffinity is not safe with respect to concurrent cpu hotplug and cpusets changes. This new "cpu pairing" concept addresses those limitations.

The key idea behind "sched_pair_cpu" is to let a worker thread eat CPU time on the target CPU while the user-space thread is provided access to user-space per-cpu data of that CPU, even though it is running on another CPU. If any of the worker or the user-space thread is preempted, it preempts the paired thread through an IPI. The priority of the per-cpu worker threads can be quite low, and is configurable by system administrators.

It provides to user-space threads access to user-space per-CPU data structures of the target CPU without requiring the task to migrate to that CPU, thus without cache pollution which would be detrimental to use-cases where the cgroup cpusets are used to partition the system.

While paired with a target CPU, restartable sequences can be used to update the per-cpu data safely against preemption on that CPU.

CPU hotplug offline CPUs are handled by the worker thread as well: it acts as a queue manager, allowing a single work item to progress at a time for the offlined CPU which it handles.

For instance, this allows implementing this userspace library API for incrementing a per-cpu counter for a specific cpu number received as parameter:

static inline __attribute__((always_inline)) int percpu_addv(intptr_t *v, intptr_t count, int cpu)

{
        int ret;

        ret = rseq_addv(v, count, cpu);
check:
        if (rseq_unlikely(ret)) {
                sched_pair_cpu_set(cpu);
                ret = rseq_addv(v, count, percpu_current_cpu());
                sched_pair_cpu_clear();
                goto check;
        }
        return 0;
}
CC Thomas Gleixner <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Joel Fernandes <[email protected]>
CC "H . Peter Anvin" <[email protected]>
CC Paul Turner <[email protected]>
CC Boqun Feng <[email protected]>
CC Steven Rostedt <[email protected]>
CC Andy Lutomirski <[email protected]>
CC "Paul E. McKenney" <[email protected]>
CC Julien Desfossez <[email protected]>

---
Changes since v1:
- Update commit message,
- Clear pairing on execve.
- Introduce CONFIG_SCHED_PAIR_CPU and CONFIG_HAVE_SCHED_PAIR_CPU.
- Add missing memory barrier before returning to user-space.
- Fixes following stress-testing and tracing.
- Introduce worker timeout (4ms).

---
 arch/Kconfig                           |   6 +
 arch/x86/Kconfig                       |   1 +
 arch/x86/entry/common.c                |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 fs/exec.c                              |   1 +
 include/linux/kthread.h                |  25 --
 include/linux/sched.h                  | 100 ++++++
 include/linux/syscalls.h               |   1 +
 include/uapi/asm-generic/unistd.h      |   4 +-
 include/uapi/linux/sched.h             |   6 +
 init/Kconfig                           |  12 +
 init/init_task.c                       |   3 +
 kernel/fork.c                          |   1 +
 kernel/sched/core.c                    | 425 +++++++++++++++++++++++++
 kernel/sys_ni.c                        |   1 +
 16 files changed, 563 insertions(+), 26 deletions(-)
arch/Kconfig +6 -0
index 8cc35dc556c7..95f71db709a0 100644
@@ -321,6 +321,12 @@ config HAVE_RSEQ
 	  This symbol should be selected by an architecture if it
 	  supports an implementation of restartable sequences.
 
+config HAVE_SCHED_PAIR_CPU
+	bool
+	help
+	  This symbol should be selected by an architecture if it
+	  supports an implementation of the sched_pair_cpu system call.
+
 config HAVE_FUNCTION_ARG_ACCESS_API
 	bool
 	help
arch/x86/Kconfig +1 -0
index 6a0cc524882d..b59145da4cae 100644
@@ -214,6 +214,7 @@ config X86
 	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
 	select HAVE_STACK_VALIDATION		if X86_64
 	select HAVE_RSEQ
+	select HAVE_SCHED_PAIR_CPU
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UNSTABLE_SCHED_CLOCK
 	select HAVE_USER_RETURN_NOTIFIER
arch/x86/entry/common.c +1 -0
index bd3f14175193..18e05eff6d12 100644
@@ -215,6 +215,7 @@ static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags)
 			clear_thread_flag(TIF_NOTIFY_RESUME);
 			tracehook_notify_resume(regs);
 			rseq_handle_notify_resume(NULL, regs);
+			sched_pair_cpu_handle_notify_resume(NULL, regs);
 		}
 
 		if (cached_flags & _TIF_USER_RETURN_NOTIFY)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
index d8f8a1a69ed1..3093ba6c06b3 100644
@@ -443,3 +443,4 @@
 437	i386	openat2			sys_openat2
 438	i386	pidfd_getfd		sys_pidfd_getfd
 439	i386	faccessat2		sys_faccessat2
+440	i386	sched_pair_cpu		sys_sched_pair_cpu
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index 78847b32e137..ebff2953ddbe 100644
@@ -360,6 +360,7 @@
 437	common	openat2			sys_openat2
 438	common	pidfd_getfd		sys_pidfd_getfd
 439	common	faccessat2		sys_faccessat2
+440	common	sched_pair_cpu		sys_sched_pair_cpu
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
fs/exec.c +1 -0
index e6e8a9a70327..0316d58d2013 100644
@@ -1931,6 +1931,7 @@ static int __do_execve_file(int fd, struct filename *filename,
 	current->fs->in_exec = 0;
 	current->in_execve = 0;
 	rseq_execve(current);
+	sched_pair_cpu_execve();
 	acct_update_integrals(current);
 	task_numa_free(current, false);
 	free_bprm(bprm);
include/linux/kthread.h +0 -25
index 65b81e0c494d..0c330e7c1453 100644
@@ -78,31 +78,6 @@ extern int tsk_fork_get_node(struct task_struct *tsk);
  * respectively.  Queued kthread_works are processed by a kthread
  * running kthread_worker_fn().
  */
-struct kthread_work;
-typedef void (*kthread_work_func_t)(struct kthread_work *work);
-void kthread_delayed_work_timer_fn(struct timer_list *t);
-
-enum {
-	KTW_FREEZABLE		= 1 << 0,	/* freeze during suspend */
-};
-
-struct kthread_worker {
-	unsigned int		flags;
-	raw_spinlock_t		lock;
-	struct list_head	work_list;
-	struct list_head	delayed_work_list;
-	struct task_struct	*task;
-	struct kthread_work	*current_work;
-};
-
-struct kthread_work {
-	struct list_head	node;
-	kthread_work_func_t	func;
-	struct kthread_worker	*worker;
-	/* Number of canceling calls that are running at the moment. */
-	int			canceling;
-};
-
 struct kthread_delayed_work {
 	struct kthread_work work;
 	struct timer_list timer;
include/linux/sched.h +100 -0

118 lines not on this page · raw message

include/linux/syscalls.h +1 -0
index 7c354c2955f5..c567087e636d 100644
@@ -1005,6 +1005,7 @@ asmlinkage long sys_pidfd_send_signal(int pidfd, int sig,
 				       siginfo_t __user *info,
 				       unsigned int flags);
 asmlinkage long sys_pidfd_getfd(int pidfd, int fd, unsigned int flags);
+asmlinkage long sys_sched_pair_cpu(int cmd, int flags, int cpu);
 
 /*
  * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +3 -1
index f4a01305d9a6..49872b9733f3 100644
@@ -857,9 +857,11 @@ __SYSCALL(__NR_openat2, sys_openat2)
 __SYSCALL(__NR_pidfd_getfd, sys_pidfd_getfd)
 #define __NR_faccessat2 439
 __SYSCALL(__NR_faccessat2, sys_faccessat2)
+#define __NR_sched_pair_cpu 440
+__SYSCALL(__NR_sched_pair_cpu, sys_sched_pair_cpu)
 
 #undef __NR_syscalls
-#define __NR_syscalls 440
+#define __NR_syscalls 441
 
 /*
  * 32 bit systems traditionally used different
include/uapi/linux/sched.h +6 -0
index 3bac0a8ceab2..0e2c413fb280 100644
@@ -145,4 +145,10 @@ struct clone_args {
 			 SCHED_FLAG_KEEP_ALL		| \
 			 SCHED_FLAG_UTIL_CLAMP)
 
+enum sched_pair_cpu_cmd {
+	SCHED_PAIR_CPU_CMD_QUERY			= 0,
+	SCHED_PAIR_CPU_CMD_SET				= (1 << 0),
+	SCHED_PAIR_CPU_CMD_CLEAR			= (1 << 1),
+};
+
 #endif /* _UAPI_LINUX_SCHED_H */
init/Kconfig +12 -0
index a46aa8f3174d..c9c57fa3f1b9 100644
@@ -1709,6 +1709,18 @@ config RSEQ
 
 	  If unsure, say Y.
 
+config SCHED_PAIR_CPU
+	bool "Enable sched_pair_cpu() system call" if EXPERT
+	default y
+	depends on HAVE_SCHED_PAIR_CPU
+	depends on SMP
+	select RSEQ
+	help
+	  Enable the CPU pairing system call. It provides user-space
+	  a mechanism to update per-cpu data belonging to specific CPUs
+	  with restartable sequences without requiring migration of the
+	  user-space thread to that CPU.
+
 config DEBUG_RSEQ
 	default n
 	bool "Enabled debugging of rseq() system call" if EXPERT
init/init_task.c +3 -0
index 15089d15010a..6d4fbd417569 100644
@@ -96,6 +96,9 @@ struct task_struct init_task
 	.tasks		= LIST_HEAD_INIT(init_task.tasks),
 #ifdef CONFIG_SMP
 	.pushable_tasks	= PLIST_NODE_INIT(init_task.pushable_tasks, MAX_PRIO),
+#ifdef CONFIG_SCHED_PAIR_CPU
+	.pair_cpu	= -1,
+#endif
 #endif
 #ifdef CONFIG_CGROUP_SCHED
 	.sched_task_group = &root_task_group,
kernel/fork.c +1 -0
index 142b23645d82..bf615a4c4ee8 100644
@@ -2238,6 +2238,7 @@ static __latent_entropy struct task_struct *copy_process(
 	copy_seccomp(p);
 
 	rseq_fork(p, clone_flags);
+	sched_pair_cpu_fork(p, clone_flags);
 
 	/* Don't start children in a dying pid namespace */
 	if (unlikely(!(ns_of_pid(pid)->pid_allocated & PIDNS_ADDING))) {
kernel/sched/core.c +425 -0

473 lines not on this page · raw message

kernel/sys_ni.c +1 -0
index 3b69a560a7ac..54dd5e7af53a 100644
@@ -472,3 +472,4 @@ COND_SYSCALL(setuid16);
 
 /* restartable sequence */
 COND_SYSCALL(rseq);
+COND_SYSCALL(sched_pair_cpu);
Signature · 1 line
2.17.1
#1 lore raw
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]

Hi Mathieu,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on linux/master linus/master v5.8-rc1] [cannot apply to tip/x86/asm next-20200618]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/sched_pair_cpu-Introduce-scheduler-task-pairing-system-call/20200620-042752
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 87e867b4269f29dac8190bca13912d08163a277f
config: s390-randconfig-r005-20200619 (attached as .config)
compiler: clang version 11.0.0 (https://github.com/llvm/llvm-project 63700971ac9cdf198faa4a3a7c226fa579e49206)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install s390 cross compiling tool for clang build
# apt-get install binutils-s390-linux-gnu
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=s390

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> <stdin>:1523:2: warning: syscall sched_pair_cpu not implemented [-W#warnings]

#warning syscall sched_pair_cpu not implemented
^
1 warning generated.
--

>> <stdin>:1523:2: warning: syscall sched_pair_cpu not implemented [-W#warnings]

#warning syscall sched_pair_cpu not implemented
^
1 warning generated.
--
In file included from arch/s390/include/asm/diag.h:12:
In file included from include/linux/if_ether.h:19:
In file included from include/linux/skbuff.h:31:
In file included from include/linux/dma-mapping.h:11:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:72:
include/asm-generic/io.h:490:45: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] val = __le32_to_cpu(__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:119:21: note: expanded from macro '__swab32'
___constant_swab32(x) : ^
include/uapi/linux/swab.h:21:12: note: expanded from macro '___constant_swab32'
(((__u32)(x) & (__u32)0x00ff0000UL) >> 8) | ^
In file included from arch/s390/kernel/asm-offsets.c:11:
In file included from include/linux/kvm_host.h:32:
In file included from include/linux/kvm_para.h:5:
In file included from include/uapi/linux/kvm_para.h:36:
In file included from arch/s390/include/asm/kvm_para.h:25:
In file included from arch/s390/include/asm/diag.h:12:
In file included from include/linux/if_ether.h:19:
In file included from include/linux/skbuff.h:31:
In file included from include/linux/dma-mapping.h:11:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:72:
include/asm-generic/io.h:490:45: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] val = __le32_to_cpu(__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:119:21: note: expanded from macro '__swab32'
___constant_swab32(x) : ^
include/uapi/linux/swab.h:22:12: note: expanded from macro '___constant_swab32'
(((__u32)(x) & (__u32)0xff000000UL) >> 24)))
^
In file included from arch/s390/kernel/asm-offsets.c:11:
In file included from include/linux/kvm_host.h:32:
In file included from include/linux/kvm_para.h:5:
In file included from include/uapi/linux/kvm_para.h:36:
In file included from arch/s390/include/asm/kvm_para.h:25:
In file included from arch/s390/include/asm/diag.h:12:
In file included from include/linux/if_ether.h:19:
In file included from include/linux/skbuff.h:31:
In file included from include/linux/dma-mapping.h:11:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:72:
include/asm-generic/io.h:490:45: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] val = __le32_to_cpu(__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:120:12: note: expanded from macro '__swab32'
__fswab32(x))
^
In file included from arch/s390/kernel/asm-offsets.c:11:
In file included from include/linux/kvm_host.h:32:
In file included from include/linux/kvm_para.h:5:
In file included from include/uapi/linux/kvm_para.h:36:
In file included from arch/s390/include/asm/kvm_para.h:25:
In file included from arch/s390/include/asm/diag.h:12:
In file included from include/linux/if_ether.h:19:
In file included from include/linux/skbuff.h:31:
In file included from include/linux/dma-mapping.h:11:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:72:
include/asm-generic/io.h:501:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] __raw_writeb(value, PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:511:46: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] __raw_writew(cpu_to_le16(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:521:46: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] __raw_writel(cpu_to_le32(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:609:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] readsb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:617:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] readsw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:625:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] readsl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:634:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] writesb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:643:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] writesw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:652:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic] writesl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
20 warnings generated.

>> <stdin>:1523:2: warning: syscall sched_pair_cpu not implemented [-W#warnings]

#warning syscall sched_pair_cpu not implemented
^
1 warning generated.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#2 lore raw
In reply to Mathieu Desnoyers
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in https://git-scm.com/docs/git-format-patch]

Hi Mathieu,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on linux/master linus/master v5.8-rc1] [cannot apply to tip/x86/asm next-20200618]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/sched_pair_cpu-Introduce-scheduler-task-pairing-system-call/20200620-042752
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 87e867b4269f29dac8190bca13912d08163a277f
config: mips-randconfig-r004-20200619 (attached as .config)
compiler: mipsel-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=mips

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> <stdin>:1523:2: warning: #warning syscall sched_pair_cpu not implemented [-Wcpp]
--
>> <stdin>:1523:2: warning: #warning syscall sched_pair_cpu not implemented [-Wcpp]

--
arch/mips/kernel/asm-offsets.c:26:6: warning: no previous prototype for 'output_ptreg_defines' [-Wmissing-prototypes] 26 | void output_ptreg_defines(void)
| ^~~~~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:78:6: warning: no previous prototype for 'output_task_defines' [-Wmissing-prototypes] 78 | void output_task_defines(void)
| ^~~~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:93:6: warning: no previous prototype for 'output_thread_info_defines' [-Wmissing-prototypes] 93 | void output_thread_info_defines(void)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:110:6: warning: no previous prototype for 'output_thread_defines' [-Wmissing-prototypes] 110 | void output_thread_defines(void)
| ^~~~~~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:181:6: warning: no previous prototype for 'output_mm_defines' [-Wmissing-prototypes] 181 | void output_mm_defines(void)
| ^~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:220:6: warning: no previous prototype for 'output_sc_defines' [-Wmissing-prototypes] 220 | void output_sc_defines(void)
| ^~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:255:6: warning: no previous prototype for 'output_signal_defined' [-Wmissing-prototypes] 255 | void output_signal_defined(void)
| ^~~~~~~~~~~~~~~~~~~~~
arch/mips/kernel/asm-offsets.c:334:6: warning: no previous prototype for 'output_pm_defines' [-Wmissing-prototypes] 334 | void output_pm_defines(void)
| ^~~~~~~~~~~~~~~~~

>> <stdin>:1523:2: warning: #warning syscall sched_pair_cpu not implemented [-Wcpp]

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#3 lore raw

[sched_pair_cpu] e9f2fb8893: will-it-scale.per_thread_ops -67.5% regression

In reply to Mathieu Desnoyers
FYI, we noticed a -67.5% regression of will-it-scale.per_thread_ops due to commit:

Greeting,

FYI, we noticed a -67.5% regression of will-it-scale.per_thread_ops due to commit:

commit: e9f2fb8893b705bcabebf3c62b663e030faf1f4a ("[RFC PATCH v2] sched_pair_cpu: Introduce scheduler task pairing system call")
url: https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/sched_pair_cpu-Introduce-scheduler-task-pairing-system-call/20200620-042752

in testcase: will-it-scale
on test machine: 192 threads Cooper Lake with 128G memory with following parameters:

	nr_task: 100%
	mode: thread
	test: context_switch1
	cpufreq_governor: performance
	ucode: 0x86000017

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -37.5% regression |
| test machine     | 104 threads Skylake with 192G memory                          |
| test parameters  | cpufreq_governor=performance                                  |
|                  | mode=thread                                                   |
|                  | nr_task=100%                                                  |
|                  | test=context_switch1                                          |
|                  | ucode=0x2000065                                               |
+------------------+---------------------------------------------------------------+

If you fix the issue, kindly add following tag

Details are as below:
-------------------------------------------------------------------------------------------------->

To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-cpx-4s1/context_switch1/will-it-scale/0x86000017

commit:
87e867b426 ("sched/pelt: Cleanup PELT divider")
e9f2fb8893 ("sched_pair_cpu: Introduce scheduler task pairing system call")

87e867b4269f29da e9f2fb8893b705bcabebf3c62b6
---------------- ---------------------------

         %stddev     %change         %stddev
             \          |                \  
    381109           -67.5%     123961        will-it-scale.per_thread_ops
  73173098           -67.5%   23800625        will-it-scale.workload
    629488 ± 17%     +58.6%     998073 ± 42%  cpuidle.C1E.usage
    399957 ± 69%     -66.4%     134211 ±162%  numa-numastat.node2.local_node
    412420 ± 67%     -59.9%     165247 ±131%  numa-numastat.node2.numa_hit
     89.50            +6.4%      95.25        vmstat.cpu.sy
  72482192           -67.6%   23500061        vmstat.system.cs
      0.00 ±117%      -0.0        0.00 ±173%  mpstat.cpu.all.iowait%
      0.00 ± 48%      +0.0        0.00 ± 35%  mpstat.cpu.all.soft%
      9.34            -6.5        2.82        mpstat.cpu.all.usr%
    672603 ±  5%      -9.7%     607172 ±  9%  meminfo.AnonPages
     48453           +27.4%      61725        meminfo.Inactive
     48054           +27.6%      61327        meminfo.Inactive(anon)
     49405 ±  2%     +27.7%      63074        meminfo.Mapped
    205187 ±  2%     +28.0%     262594 ±  3%  meminfo.Shmem
    746.00 ± 12%     +21.7%     908.00 ±  4%  slabinfo.khugepaged_mm_slot.active_objs
    746.00 ± 12%     +21.7%     908.00 ±  4%  slabinfo.khugepaged_mm_slot.num_objs
    165639           +40.7%     233094        slabinfo.kmalloc-64.active_objs
      2590           +40.7%       3646        slabinfo.kmalloc-64.active_slabs
    165819           +40.7%     233344        slabinfo.kmalloc-64.num_objs
      2590           +40.7%       3646        slabinfo.kmalloc-64.num_slabs
      8840 ±  5%     -28.4%       6331 ± 15%  numa-vmstat.node2.nr_slab_reclaimable
    113913 ± 11%     +31.6%     149917        numa-vmstat.node2.numa_other
     65522 ±  7%     +23.1%      80627 ± 13%  numa-vmstat.node3.nr_active_anon
    125181           +11.6%     139689 ±  2%  numa-vmstat.node3.nr_file_pages
      3470 ± 20%     +87.3%       6500 ±  3%  numa-vmstat.node3.nr_inactive_anon
      6154 ±  2%     +20.8%       7437 ±  5%  numa-vmstat.node3.nr_kernel_stack
      3934 ± 12%     +50.3%       5912        numa-vmstat.node3.nr_mapped
     42101 ±  2%     +33.2%      56082 ±  3%  numa-vmstat.node3.nr_shmem
     65522 ±  7%     +23.1%      80627 ± 13%  numa-vmstat.node3.nr_zone_active_anon
      3470 ± 20%     +87.3%       6500 ±  3%  numa-vmstat.node3.nr_zone_inactive_anon
     35363 ±  5%     -28.4%      25329 ± 15%  numa-meminfo.node2.KReclaimable
     35363 ±  5%     -28.4%      25329 ± 15%  numa-meminfo.node2.SReclaimable
    262207 ±  7%     +23.0%     322429 ± 13%  numa-meminfo.node3.Active
    262207 ±  7%     +23.0%     322429 ± 13%  numa-meminfo.node3.Active(anon)
    501055           +11.5%     558705 ±  2%  numa-meminfo.node3.FilePages
     14049 ± 22%     +84.6%      25936 ±  2%  numa-meminfo.node3.Inactive
     13989 ± 23%     +85.4%      25936 ±  2%  numa-meminfo.node3.Inactive(anon)
      6154 ±  2%     +20.9%       7438 ±  5%  numa-meminfo.node3.KernelStack
     15606 ± 14%     +49.0%      23254 ±  2%  numa-meminfo.node3.Mapped
    985437 ±  2%     +17.7%    1159682 ±  8%  numa-meminfo.node3.MemUsed
    168735 ±  2%     +32.9%     224275 ±  3%  numa-meminfo.node3.Shmem
    168135 ±  5%      -9.8%     151702 ±  9%  proc-vmstat.nr_anon_pages
    388347            +3.7%     402665        proc-vmstat.nr_file_pages
     11913           +28.7%      15327        proc-vmstat.nr_inactive_anon
     32244            +9.5%      35299        proc-vmstat.nr_kernel_stack
     12356 ±  2%     +28.3%      15853        proc-vmstat.nr_mapped
     51267 ±  2%     +28.1%      65649 ±  3%  proc-vmstat.nr_shmem
     29556            +1.8%      30086        proc-vmstat.nr_slab_reclaimable
     80332            +3.3%      82953        proc-vmstat.nr_slab_unreclaimable
     11913           +28.7%      15327        proc-vmstat.nr_zone_inactive_anon
   1007396            +1.9%    1026170        proc-vmstat.numa_hit
    914187            +2.1%     932960        proc-vmstat.numa_local
     60211 ±  3%     +39.0%      83671 ±  3%  proc-vmstat.pgactivate
   1010160            +7.5%    1085798 ±  4%  proc-vmstat.pgfree
      0.17 ± 99%    +975.0%       1.79 ± 42%  sched_debug.cfs_rq:/.load_avg.min
    132891 ±  9%     +30.1%     172957 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.14 ±  6%     +27.5%       0.18 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    -33482          +404.5%    -168935        sched_debug.cfs_rq:/.spread0.avg
    196291 ± 15%     -82.8%      33829 ± 82%  sched_debug.cfs_rq:/.spread0.max
  -1178050           +78.9%   -2107947        sched_debug.cfs_rq:/.spread0.min
    132832 ±  9%     +29.8%     172431 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    581.51           +17.2%     681.44        sched_debug.cfs_rq:/.util_est_enqueued.avg
      2094 ±  4%     +70.7%       3574 ±  3%  sched_debug.cpu.avg_idle.min
     37.62 ±  2%     +63.8%      61.62 ±  2%  sched_debug.cpu.clock.stddev
     37.62 ±  2%     +63.8%      61.62 ±  2%  sched_debug.cpu.clock_task.stddev
      7029 ±  5%     +11.3%       7822        sched_debug.cpu.curr->pid.max
  56823172           -67.4%   18533495        sched_debug.cpu.nr_switches.avg
  62261802 ±  2%     -66.7%   20760559        sched_debug.cpu.nr_switches.max
  52691982           -72.7%   14402197        sched_debug.cpu.nr_switches.min
     17.36 ±  3%     +12.0%      19.44 ±  3%  sched_debug.cpu.nr_uninterruptible.stddev
 1.176e+11           -57.9%  4.952e+10        perf-stat.i.branch-instructions
      1.14            -0.2        0.98        perf-stat.i.branch-miss-rate%
 1.324e+09           -63.8%  4.798e+08        perf-stat.i.branch-misses
      0.14 ±  5%      +8.1        8.23 ±  2%  perf-stat.i.cache-miss-rate%
   1925606 ±  8%   +4653.6%   91535573        perf-stat.i.cache-misses
  2.62e+09 ±  8%     -57.3%  1.119e+09 ±  2%  perf-stat.i.cache-references
  72846121           -67.6%   23638038        perf-stat.i.context-switches
      1.13          +185.5%       3.22        perf-stat.i.cpi
 6.637e+11           +14.4%   7.59e+11        perf-stat.i.cpu-cycles
    404127 ± 11%     -98.0%       8282        perf-stat.i.cycles-between-cache-misses
      0.00 ±  6%      +0.0        0.00 ± 76%  perf-stat.i.dTLB-load-miss-rate%
 1.707e+11           -61.8%  6.516e+10        perf-stat.i.dTLB-loads
      0.00            +0.0        0.00 ±  9%  perf-stat.i.dTLB-store-miss-rate%
    114494 ±  2%     -30.8%      79221 ±  8%  perf-stat.i.dTLB-store-misses
 1.046e+11           -61.5%  4.032e+10        perf-stat.i.dTLB-stores
 4.538e+08           -59.7%  1.827e+08        perf-stat.i.iTLB-load-misses
 5.902e+11           -60.1%  2.357e+11        perf-stat.i.instructions
      0.89           -64.9%       0.31        perf-stat.i.ipc
      3.46           +14.5%       3.96        perf-stat.i.metric.GHz
      1.44           -57.0%       0.62 ±  8%  perf-stat.i.metric.K/sec
      2061           -60.4%     815.41        perf-stat.i.metric.M/sec
    298276 ±  4%   +2620.5%    8114651        perf-stat.i.node-load-misses
     47637 ± 36%   +1216.0%     626889        perf-stat.i.node-loads
     92.96           -11.0       81.95        perf-stat.i.node-store-miss-rate%
     87054 ±  7%   +1416.2%    1319891        perf-stat.i.node-store-misses
     10937 ± 12%   +2569.1%     291918        perf-stat.i.node-stores
      1.13            -0.2        0.97        perf-stat.overall.branch-miss-rate%
      0.07 ± 12%      +8.1        8.18 ±  2%  perf-stat.overall.cache-miss-rate%
      1.12          +186.1%       3.22        perf-stat.overall.cpi
    349097 ±  8%     -97.6%       8304        perf-stat.overall.cycles-between-cache-misses
      0.00 ±  8%      +0.0        0.00 ± 71%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      +0.0        0.00 ±  7%  perf-stat.overall.dTLB-store-miss-rate%
      0.89           -65.0%       0.31        perf-stat.overall.ipc
     85.95 ±  5%      +6.8       92.78        perf-stat.overall.node-load-miss-rate%
     88.64            -7.0       81.67        perf-stat.overall.node-store-miss-rate%
   2429767           +23.4%    2998949        perf-stat.overall.path-length
 1.172e+11           -57.9%   4.94e+10        perf-stat.ps.branch-instructions
 1.319e+09           -63.8%  4.782e+08        perf-stat.ps.branch-misses
   1909280 ±  8%   +4671.3%   91098325        perf-stat.ps.cache-misses
 2.609e+09 ±  8%     -57.3%  1.115e+09 ±  2%  perf-stat.ps.cache-references
  72607352           -67.5%   23581726        perf-stat.ps.context-switches
 6.615e+11           +14.4%  7.565e+11        perf-stat.ps.cpu-cycles
 1.701e+11           -61.8%    6.5e+10        perf-stat.ps.dTLB-loads
    114417 ±  2%     -30.1%      79978 ±  7%  perf-stat.ps.dTLB-store-misses
 1.043e+11           -61.4%  4.022e+10        perf-stat.ps.dTLB-stores
 4.523e+08           -59.7%  1.823e+08        perf-stat.ps.iTLB-load-misses
 5.882e+11           -60.0%  2.352e+11        perf-stat.ps.instructions
    296909 ±  3%   +2620.2%    8076510        perf-stat.ps.node-load-misses
     49167 ± 37%   +1177.6%     628163        perf-stat.ps.node-loads
     86332 ±  7%   +1424.9%    1316449        perf-stat.ps.node-store-misses
     10984 ± 12%   +2590.3%     295531        perf-stat.ps.node-stores
 1.778e+14           -59.9%  7.138e+13        perf-stat.total.instructions
     13.91 ±  2%      -7.8        6.16 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     13.78 ±  2%      -7.7        6.08 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     11.75 ±  2%      -7.7        4.07        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
     13.56 ±  2%      -7.6        6.00 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      7.27            -4.8        2.45        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      6.04            -4.0        2.08        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      4.35            -3.0        1.40        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.new_sync_read
      4.48 ±  4%      -2.8        1.69 ±  3%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      5.66 ±  5%      -2.8        2.89 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
     56.05            -2.4       53.61        perf-profile.calltrace.cycles-pp.__GI___libc_read
      3.38            -2.4        0.97        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      3.10            -2.2        0.89        perf-profile.calltrace.cycles-pp.switch_fpu_return.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      2.94            -2.0        0.91        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
      2.68            -1.9        0.81        perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      2.80            -1.7        1.06 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.82            -1.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.__switch_to_asm.__GI___libc_read
      2.09            -1.5        0.57 ±  3%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
      2.09            -1.5        0.63 ±  3%  perf-profile.calltrace.cycles-pp.__switch_to.__GI___libc_read
      1.98 ±  2%      -1.4        0.55        perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      2.08            -1.4        0.73        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      2.06 ±  6%      -1.2        0.82 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.70            -1.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      1.92            -1.1        0.80 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.53 ±  5%      -1.0        0.52 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.39            -1.0        0.39 ± 57%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      2.12 ±  6%      -0.9        1.25 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      1.42            -0.8        0.64        perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.72 ±  2%      -0.7        0.99 ±  2%  perf-profile.calltrace.cycles-pp.native_write_msr.__GI___libc_read
      1.00 ± 13%      -0.7        0.28 ±100%  perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task
      1.23 ±  2%      -0.7        0.55        perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.ring_buffer_lock_reserve.trace_vbprintk.__trace_bprintk.__schedule.schedule
      0.00            +1.2        1.21 ±  2%  perf-profile.calltrace.cycles-pp.format_decode.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule
      0.00            +1.7        1.69 ±  2%  perf-profile.calltrace.cycles-pp.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule.schedule
     43.48            +2.3       45.82        perf-profile.calltrace.cycles-pp.__GI___libc_write
     47.40            +3.4       50.76        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     46.70            +3.8       50.53        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     40.47            +4.6       45.12        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.00            +4.6        4.65 ±  2%  perf-profile.calltrace.cycles-pp.trace_vbprintk.__trace_bprintk.__schedule.schedule.pipe_read
      0.00            +4.7        4.71 ±  2%  perf-profile.calltrace.cycles-pp.__trace_bprintk.__schedule.schedule.pipe_read.new_sync_read
     40.14            +4.9       45.03        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     38.64            +5.9       44.52        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     36.98            +7.0       43.95        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     42.04            +7.2       49.19        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      0.00            +8.0        8.02        perf-profile.calltrace.cycles-pp.probe_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      0.00            +8.1        8.06        perf-profile.calltrace.cycles-pp.probe_sched_wakeup.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
     40.24            +8.4       48.67        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     34.42            +8.7       43.09        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.75            +9.1       42.86        perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     36.58           +11.0       47.62        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     34.79           +12.5       47.26        perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
     27.84           +13.2       41.02        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     26.69           +13.9       40.55        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
     26.46           +14.0       40.47        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
     26.05           +14.3       40.35        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
     22.82           +21.0       43.78        perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
     22.05           +21.6       43.61        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
      0.00           +22.0       21.99        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.00           +22.0       22.05        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00           +23.3       23.30        perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read
      0.00           +23.4       23.35        perf-profile.calltrace.cycles-pp.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read.new_sync_read
      1.71           +29.0       30.73        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     13.91 ±  2%      -7.8        6.16 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
     11.79 ±  2%      -7.7        4.08        perf-profile.children.cycles-pp.dequeue_task_fair
     13.80 ±  2%      -7.7        6.09 ±  2%  perf-profile.children.cycles-pp.activate_task
     13.59 ±  2%      -7.6        6.00 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      8.53 ±  3%      -5.1        3.44 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      7.29            -4.8        2.46        perf-profile.children.cycles-pp.select_task_rq_fair
      7.02            -4.4        2.62        perf-profile.children.cycles-pp.update_curr
      6.21            -4.1        2.12        perf-profile.children.cycles-pp.select_idle_sibling
      5.56            -3.7        1.90 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      4.62            -3.2        1.46        perf-profile.children.cycles-pp.pick_next_task_fair
      5.79 ±  4%      -2.9        2.92 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      4.56 ±  4%      -2.9        1.71 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      3.87            -2.7        1.16        perf-profile.children.cycles-pp.__prepare_exit_to_usermode
     56.06            -2.3       53.76        perf-profile.children.cycles-pp.__GI___libc_read
      3.13            -2.2        0.89        perf-profile.children.cycles-pp.switch_fpu_return
      3.05            -2.1        0.94        perf-profile.children.cycles-pp.prepare_to_wait_event
      2.93            -2.0        0.89 ±  2%  perf-profile.children.cycles-pp.__switch_to
      2.55            -1.9        0.70 ±  2%  perf-profile.children.cycles-pp.mutex_lock
      2.87 ±  8%      -1.8        1.09 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      2.46 ±  3%      -1.7        0.74 ±  3%  perf-profile.children.cycles-pp.security_file_permission
      2.68            -1.6        1.07        perf-profile.children.cycles-pp.update_cfs_group
      2.40            -1.6        0.80        perf-profile.children.cycles-pp.sched_clock_cpu
      2.26            -1.6        0.69        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      2.13            -1.5        0.58 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
      2.09            -1.5        0.59 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      2.04 ±  2%      -1.5        0.57        perf-profile.children.cycles-pp.copy_page_from_iter
      2.23            -1.4        0.78 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      2.04 ±  8%      -1.4        0.63        perf-profile.children.cycles-pp.__fdget_pos
      2.16            -1.4        0.75        perf-profile.children.cycles-pp.set_next_entity
      1.92            -1.4        0.54        perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.90            -1.4        0.52 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      1.94 ±  8%      -1.3        0.60        perf-profile.children.cycles-pp.__fget_light
      1.85            -1.3        0.52 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      2.17            -1.3        0.88        perf-profile.children.cycles-pp.sched_clock
      1.80            -1.3        0.53        perf-profile.children.cycles-pp.__update_load_avg_se
      2.09            -1.2        0.85        perf-profile.children.cycles-pp.native_sched_clock
      1.74            -1.2        0.51 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      1.55            -1.1        0.45        perf-profile.children.cycles-pp.check_preempt_curr
      1.52            -1.1        0.43        perf-profile.children.cycles-pp.update_rq_clock
      1.57 ±  5%      -1.1        0.49 ±  5%  perf-profile.children.cycles-pp.common_file_perm
      1.43 ± 12%      -1.0        0.43 ±  3%  perf-profile.children.cycles-pp.__fget_files
      1.50            -1.0        0.53 ±  3%  perf-profile.children.cycles-pp.cpumask_next_wrap
      1.50            -1.0        0.53 ±  2%  perf-profile.children.cycles-pp._find_next_bit
      1.33 ±  2%      -0.9        0.38        perf-profile.children.cycles-pp.fsnotify
      1.30            -0.9        0.36        perf-profile.children.cycles-pp.check_preempt_wakeup
      1.90 ±  2%      -0.8        1.06 ±  3%  perf-profile.children.cycles-pp.native_write_msr
      1.02            -0.7        0.28        perf-profile.children.cycles-pp.__might_sleep
      1.27            -0.7        0.58 ±  4%  perf-profile.children.cycles-pp.update_min_vruntime
      0.98 ±  3%      -0.7        0.29 ±  2%  perf-profile.children.cycles-pp.touch_atime
      0.83            -0.6        0.19 ±  2%  perf-profile.children.cycles-pp._cond_resched
      0.94 ±  5%      -0.6        0.31 ±  4%  perf-profile.children.cycles-pp.current_time
      0.86            -0.6        0.24 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.89            -0.6        0.27 ±  2%  perf-profile.children.cycles-pp.pick_next_entity
      0.85 ±  2%      -0.6        0.25        perf-profile.children.cycles-pp.fsnotify_parent
      0.86            -0.6        0.26        perf-profile.children.cycles-pp.mutex_unlock
      0.85            -0.6        0.25 ±  3%  perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      0.83            -0.6        0.23 ±  2%  perf-profile.children.cycles-pp.cpumask_next
      0.82            -0.6        0.23 ±  2%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.84            -0.6        0.26        perf-profile.children.cycles-pp.account_entity_dequeue
      0.82 ±  2%      -0.6        0.24 ±  2%  perf-profile.children.cycles-pp.atime_needs_update
      0.75 ± 13%      -0.6        0.20 ± 12%  perf-profile.children.cycles-pp.cpuacct_charge
      0.75            -0.5        0.21 ±  2%  perf-profile.children.cycles-pp.copyout
      0.78 ±  2%      -0.5        0.26 ±  2%  perf-profile.children.cycles-pp.___perf_sw_event
      0.76            -0.5        0.26 ±  3%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.71            -0.5        0.21 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.69            -0.5        0.20        perf-profile.children.cycles-pp.copyin
      0.68            -0.5        0.21 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.79            -0.5        0.33 ±  5%  perf-profile.children.cycles-pp.set_next_buddy
      0.62 ±  2%      -0.4        0.19 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.85 ±  6%      -0.4        0.42 ±  6%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.88 ±  6%      -0.4        0.46 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.80 ±  5%      -0.4        0.38 ±  6%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.80 ±  6%      -0.4        0.38 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.52            -0.4        0.12        perf-profile.children.cycles-pp.rcu_all_qs
      0.64            -0.4        0.26 ±  3%  perf-profile.children.cycles-pp.__pthread_enable_asynccancel
      0.53            -0.4        0.15 ±  2%  perf-profile.children.cycles-pp.finish_wait
      0.82            -0.4        0.45 ±  2%  perf-profile.children.cycles-pp.finish_task_switch
      0.73 ±  4%      -0.4        0.37        perf-profile.children.cycles-pp.file_update_time
      0.77 ±  5%      -0.4        0.41 ±  6%  perf-profile.children.cycles-pp.asm_call_on_stack
      0.55            -0.3        0.20 ±  2%  perf-profile.children.cycles-pp.fput_many
      0.46 ±  2%      -0.3        0.13 ±  3%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.47 ±  4%      -0.3        0.17 ±  8%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.41            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.39            -0.3        0.12 ±  3%  perf-profile.children.cycles-pp.clear_buddies
      0.34 ±  2%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.35 ± 11%      -0.2        0.11 ±  3%  perf-profile.children.cycles-pp.__sb_start_write
      0.32 ±  2%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.wakeup_preempt_entity
      0.31 ±  2%      -0.2        0.09        perf-profile.children.cycles-pp.iov_iter_init
      0.30            -0.2        0.08 ±  5%  perf-profile.children.cycles-pp.put_prev_entity
      0.27 ±  9%      -0.2        0.07        perf-profile.children.cycles-pp.apparmor_file_permission
      0.26 ±  7%      -0.2        0.06 ± 13%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.28 ±  3%      -0.2        0.09 ±  4%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.28            -0.2        0.09        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.24 ±  6%      -0.2        0.06        perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.28            -0.2        0.10        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.38 ±  3%      -0.2        0.21 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.23            -0.2        0.07 ±  6%  perf-profile.children.cycles-pp.__list_add_valid
      0.23 ± 13%      -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.timestamp_truncate
      0.25 ±  6%      -0.2        0.10        perf-profile.children.cycles-pp.aa_file_perm
      0.21 ±  2%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.__sb_end_write
      0.22            -0.1        0.07        perf-profile.children.cycles-pp.find_next_bit
      0.21 ±  2%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__x64_sys_read
      0.30 ±  6%      -0.1        0.16 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.20            -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.place_entity
      0.21            -0.1        0.07        perf-profile.children.cycles-pp.__x64_sys_write
      0.23 ± 13%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.ktime_get
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__bitmap_and
      0.23 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.deactivate_task
      0.20 ±  2%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.15 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.fput
      0.30 ±  2%      -0.1        0.23 ±  3%  perf-profile.children.cycles-pp.rb_insert_color
      0.20 ±  2%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.tick_sched_handle
      0.20 ±  2%      -0.1        0.13 ± 10%  perf-profile.children.cycles-pp.update_process_times
      0.15 ± 10%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      0.13 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.rb_erase
      0.13 ±  5%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.09 ±  8%      -0.0        0.06 ± 16%  perf-profile.children.cycles-pp.task_tick_fair
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ring_buffer_nest_end
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.tracing_generic_entry_update
      0.00            +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.rb_commit
      0.00            +0.2        0.18 ±  6%  perf-profile.children.cycles-pp.ring_buffer_nest_start
      0.00            +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.__rb_reserve_next
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.trace_clock_local
      0.00            +0.2        0.21 ±  2%  perf-profile.children.cycles-pp.ring_buffer_unlock_commit
      0.00            +0.6        0.60 ±  2%  perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      0.00            +1.3        1.26        perf-profile.children.cycles-pp.format_decode
      0.00            +1.7        1.74 ±  2%  perf-profile.children.cycles-pp.vbin_printf
     43.51            +2.5       46.00        perf-profile.children.cycles-pp.__GI___libc_write
      0.00            +4.7        4.70 ±  2%  perf-profile.children.cycles-pp.trace_vbprintk
      0.00            +4.7        4.72 ±  2%  perf-profile.children.cycles-pp.__trace_bprintk
     38.70            +5.9       44.55        perf-profile.children.cycles-pp.ksys_write
     37.07            +6.9       43.99        perf-profile.children.cycles-pp.vfs_write
     42.15            +7.1       49.22        perf-profile.children.cycles-pp.ksys_read
     87.93            +8.0       95.93        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +8.0        8.04        perf-profile.children.cycles-pp.probe_sched_switch
      0.00            +8.1        8.07        perf-profile.children.cycles-pp.probe_sched_wakeup
     40.35            +8.4       48.70        perf-profile.children.cycles-pp.vfs_read
     86.96            +8.7       95.63        perf-profile.children.cycles-pp.do_syscall_64
     34.46            +8.7       43.13        perf-profile.children.cycles-pp.new_sync_write
     33.82            +9.1       42.89        perf-profile.children.cycles-pp.pipe_write
     36.63           +11.0       47.64        perf-profile.children.cycles-pp.new_sync_read
     34.98           +12.3       47.32        perf-profile.children.cycles-pp.pipe_read
     27.87           +13.2       41.03        perf-profile.children.cycles-pp.__wake_up_common_lock
     26.71           +13.8       40.55        perf-profile.children.cycles-pp.__wake_up_common
     26.49           +14.0       40.48        perf-profile.children.cycles-pp.autoremove_wake_function
     26.07           +14.3       40.36        perf-profile.children.cycles-pp.try_to_wake_up
     22.95           +20.9       43.86        perf-profile.children.cycles-pp.schedule
     22.35           +21.4       43.73        perf-profile.children.cycles-pp.__schedule
      0.00           +22.1       22.07        perf-profile.children.cycles-pp.tracing_record_taskinfo
      0.00           +23.4       23.41        perf-profile.children.cycles-pp.tracing_record_taskinfo_sched_switch
      1.74           +29.0       30.74        perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00           +45.3       45.34        perf-profile.children.cycles-pp.trace_save_cmdline
      3.11            -2.2        0.89        perf-profile.self.cycles-pp.switch_fpu_return
      3.93            -2.1        1.83 ±  3%  perf-profile.self.cycles-pp.update_load_avg
      3.36 ±  3%      -2.0        1.36 ±  2%  perf-profile.self.cycles-pp.update_curr
      2.76            -1.9        0.84 ±  2%  perf-profile.self.cycles-pp.__switch_to
      2.78 ±  8%      -1.7        1.07 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      2.64            -1.6        1.06        perf-profile.self.cycles-pp.update_cfs_group
      2.17            -1.5        0.66        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.89            -1.4        0.52 ±  3%  perf-profile.self.cycles-pp.__switch_to_asm
      1.89            -1.4        0.53        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.88            -1.3        0.55 ±  3%  perf-profile.self.cycles-pp.__schedule
      1.89            -1.3        0.57 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      1.81            -1.3        0.51 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      1.75            -1.3        0.50 ±  2%  perf-profile.self.cycles-pp.pipe_read
      1.75            -1.2        0.52        perf-profile.self.cycles-pp.__update_load_avg_se
      1.72            -1.2        0.51 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      1.85            -1.2        0.64        perf-profile.self.cycles-pp.select_idle_sibling
      1.65            -1.2        0.44        perf-profile.self.cycles-pp.reweight_entity
      2.03            -1.2        0.83        perf-profile.self.cycles-pp.native_sched_clock
      2.08            -1.2        0.90 ±  3%  perf-profile.self.cycles-pp.enqueue_task_fair
      1.44            -1.2        0.28 ±  4%  perf-profile.self.cycles-pp.new_sync_read
      1.40 ± 12%      -1.0        0.42 ±  4%  perf-profile.self.cycles-pp.__fget_files
      1.39 ±  2%      -1.0        0.41 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      1.49            -1.0        0.52 ±  2%  perf-profile.self.cycles-pp._find_next_bit
      1.32 ±  7%      -0.9        0.39 ±  6%  perf-profile.self.cycles-pp.common_file_perm
      1.28 ±  2%      -0.9        0.37 ±  2%  perf-profile.self.cycles-pp.fsnotify
      1.33            -0.9        0.42        perf-profile.self.cycles-pp.prepare_to_wait_event
      1.90 ±  2%      -0.8        1.06 ±  3%  perf-profile.self.cycles-pp.native_write_msr
      1.06            -0.8        0.30 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.08            -0.7        0.34        perf-profile.self.cycles-pp.select_task_rq_fair
      1.05            -0.7        0.31 ±  3%  perf-profile.self.cycles-pp.vfs_read
      1.03            -0.7        0.32        perf-profile.self.cycles-pp.dequeue_task_fair
      1.00            -0.7        0.29 ±  2%  perf-profile.self.cycles-pp.pipe_write
      1.25            -0.7        0.57 ±  4%  perf-profile.self.cycles-pp.update_min_vruntime
      1.26 ± 10%      -0.7        0.59 ±  7%  perf-profile.self.cycles-pp.enqueue_entity
      0.97            -0.7        0.31 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.91            -0.7        0.25 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.95            -0.6        0.30        perf-profile.self.cycles-pp.mutex_lock
      0.88            -0.6        0.25        perf-profile.self.cycles-pp.pick_next_task_fair
      0.80 ±  3%      -0.6        0.21 ±  3%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.90            -0.6        0.31 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.83            -0.6        0.26        perf-profile.self.cycles-pp.mutex_unlock
      0.82            -0.6        0.25        perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      0.79 ±  2%      -0.6        0.23        perf-profile.self.cycles-pp.__GI___libc_write
      0.78            -0.6        0.23 ±  2%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.76 ±  2%      -0.6        0.21 ±  2%  perf-profile.self.cycles-pp.__GI___libc_read
      0.75 ± 13%      -0.6        0.20 ± 13%  perf-profile.self.cycles-pp.cpuacct_charge
      0.76 ±  2%      -0.5        0.23 ±  3%  perf-profile.self.cycles-pp.fsnotify_parent
      0.74 ±  3%      -0.5        0.26 ±  3%  perf-profile.self.cycles-pp.vfs_write
      0.71            -0.5        0.23        perf-profile.self.cycles-pp.account_entity_dequeue
      0.68            -0.5        0.21 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.58            -0.5        0.11 ±  4%  perf-profile.self.cycles-pp.schedule
      0.69 ±  2%      -0.5        0.23        perf-profile.self.cycles-pp.___perf_sw_event
      0.70            -0.5        0.24 ±  3%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.78            -0.5        0.33 ±  5%  perf-profile.self.cycles-pp.set_next_buddy
      0.66            -0.4        0.21 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.58            -0.4        0.16        perf-profile.self.cycles-pp.ksys_read
      0.56            -0.4        0.16 ±  2%  perf-profile.self.cycles-pp.cpumask_next_wrap
      0.51 ±  3%      -0.4        0.12 ±  3%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.53 ±  2%      -0.4        0.15 ±  5%  perf-profile.self.cycles-pp.update_rq_clock
      0.55 ±  2%      -0.4        0.17 ±  4%  perf-profile.self.cycles-pp.available_idle_cpu
      0.61            -0.4        0.25 ±  3%  perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      0.49 ±  2%      -0.4        0.13 ±  3%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.51            -0.4        0.15 ±  3%  perf-profile.self.cycles-pp.pick_next_entity
      0.54 ±  4%      -0.4        0.19 ±  6%  perf-profile.self.cycles-pp.new_sync_write
      0.52            -0.3        0.20 ±  2%  perf-profile.self.cycles-pp.fput_many
      0.42            -0.3        0.10        perf-profile.self.cycles-pp.rcu_all_qs
      0.45 ±  3%      -0.3        0.13        perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.47 ± 14%      -0.3        0.16 ±  4%  perf-profile.self.cycles-pp.__fget_light
      0.47 ±  4%      -0.3        0.17 ±  8%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.39 ± 10%      -0.3        0.11        perf-profile.self.cycles-pp.security_file_permission
      0.40            -0.3        0.12 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.38            -0.3        0.11 ±  7%  perf-profile.self.cycles-pp.__prepare_exit_to_usermode
      0.42            -0.3        0.15 ±  2%  perf-profile.self.cycles-pp.set_next_entity
      0.34            -0.3        0.07        perf-profile.self.cycles-pp._cond_resched
      0.46            -0.2        0.21 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.33 ±  8%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.atime_needs_update
      0.35 ± 11%      -0.2        0.11 ±  3%  perf-profile.self.cycles-pp.__sb_start_write
      0.33            -0.2        0.10 ±  4%  perf-profile.self.cycles-pp.clear_buddies
      0.31            -0.2        0.10        perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.28            -0.2        0.07        perf-profile.self.cycles-pp.iov_iter_init
      0.30 ±  3%      -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.wakeup_preempt_entity
      0.26 ±  8%      -0.2        0.06 ± 11%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.25 ± 10%      -0.2        0.06        perf-profile.self.cycles-pp.apparmor_file_permission
      0.26            -0.2        0.08 ±  6%  perf-profile.self.cycles-pp.autoremove_wake_function
      0.23 ±  7%      -0.2        0.06 ±  7%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.27            -0.2        0.09 ±  4%  perf-profile.self.cycles-pp.ksys_write
      0.23 ±  6%      -0.2        0.06 ±  9%  perf-profile.self.cycles-pp.current_time
      0.25            -0.2        0.08        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.26 ±  2%      -0.2        0.10 ±  5%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.21 ±  2%      -0.2        0.05        perf-profile.self.cycles-pp.sched_clock_cpu
      0.23            -0.2        0.07 ±  6%  perf-profile.self.cycles-pp.put_prev_entity
      0.23 ±  3%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.check_preempt_curr
      0.21 ±  2%      -0.2        0.06        perf-profile.self.cycles-pp.__sb_end_write
      0.22 ± 13%      -0.1        0.07        perf-profile.self.cycles-pp.timestamp_truncate
      0.32 ±  3%      -0.1        0.17 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.22 ±  3%      -0.1        0.07        perf-profile.self.cycles-pp.__wake_up_common
      0.20 ±  2%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.19 ±  2%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__x64_sys_read
      0.23 ± 15%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.ktime_get
      0.23 ±  8%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.aa_file_perm
      0.20            -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.__x64_sys_write
      0.20 ±  2%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.18 ±  2%      -0.1        0.05        perf-profile.self.cycles-pp.place_entity
      0.18 ±  2%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.23 ±  2%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.deactivate_task
      0.18 ±  6%      -0.1        0.07        perf-profile.self.cycles-pp.activate_task
      0.16 ±  5%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  2%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.27            -0.1        0.20 ±  2%  perf-profile.self.cycles-pp.file_update_time
      0.29 ±  2%      -0.1        0.22 ±  4%  perf-profile.self.cycles-pp.rb_insert_color
      0.12 ±  3%      -0.0        0.07        perf-profile.self.cycles-pp.ttwu_do_activate
      0.11 ±  3%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.rb_erase
      0.21 ±  2%      -0.0        0.17 ± 10%  perf-profile.self.cycles-pp.ttwu_do_wakeup
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.tracing_record_taskinfo_skip
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.ring_buffer_nest_end
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.trace_clock_local
      0.00            +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.tracing_generic_entry_update
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.ring_buffer_unlock_commit
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.rb_commit
      0.00            +0.2        0.18 ±  6%  perf-profile.self.cycles-pp.ring_buffer_nest_start
      0.00            +0.2        0.18 ±  4%  perf-profile.self.cycles-pp.ring_buffer_lock_reserve
      0.00            +0.2        0.19 ±  2%  perf-profile.self.cycles-pp.__rb_reserve_next
      0.00            +0.5        0.51 ±  3%  perf-profile.self.cycles-pp.vbin_printf
      0.00            +1.2        1.22 ±  2%  perf-profile.self.cycles-pp.format_decode
      0.00            +1.8        1.81 ±  9%  perf-profile.self.cycles-pp.trace_vbprintk
      0.00            +8.0        8.03        perf-profile.self.cycles-pp.probe_sched_switch
      0.00            +8.1        8.06        perf-profile.self.cycles-pp.probe_sched_wakeup
      0.00           +45.3       45.33        perf-profile.self.cycles-pp.trace_save_cmdline
    217.75 ± 14%   +2074.2%       4734 ± 89%  interrupts.33:PCI-MSI.524291-edge.eth0-TxRx-2
      9726           +14.6%      11141        interrupts.CPU0.NMI:Non-maskable_interrupts
      9726           +14.6%      11141        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      8509 ± 24%     +30.9%      11137        interrupts.CPU1.NMI:Non-maskable_interrupts
      8509 ± 24%     +30.9%      11137        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11137        interrupts.CPU10.NMI:Non-maskable_interrupts
      9727           +14.5%      11137        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      9730           +14.5%      11138        interrupts.CPU100.NMI:Non-maskable_interrupts
      9730           +14.5%      11138        interrupts.CPU100.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11140        interrupts.CPU101.NMI:Non-maskable_interrupts
      9727           +14.5%      11140        interrupts.CPU101.PMI:Performance_monitoring_interrupts
      9724           +14.6%      11141        interrupts.CPU102.NMI:Non-maskable_interrupts
      9724           +14.6%      11141        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11138        interrupts.CPU103.NMI:Non-maskable_interrupts
      9725           +14.5%      11138        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11139        interrupts.CPU104.NMI:Non-maskable_interrupts
      9725           +14.5%      11139        interrupts.CPU104.PMI:Performance_monitoring_interrupts
      9726           +14.5%      11139        interrupts.CPU105.NMI:Non-maskable_interrupts
      9726           +14.5%      11139        interrupts.CPU105.PMI:Performance_monitoring_interrupts
      9726           +14.5%      11139        interrupts.CPU106.NMI:Non-maskable_interrupts
      9726           +14.5%      11139        interrupts.CPU106.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11140        interrupts.CPU109.NMI:Non-maskable_interrupts
      9723           +14.6%      11140        interrupts.CPU109.PMI:Performance_monitoring_interrupts
    217.75 ± 14%   +2074.2%       4734 ± 89%  interrupts.CPU11.33:PCI-MSI.524291-edge.eth0-TxRx-2
      9728           +14.5%      11137        interrupts.CPU11.NMI:Non-maskable_interrupts
      9728           +14.5%      11137        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11141        interrupts.CPU110.NMI:Non-maskable_interrupts
      9723           +14.6%      11141        interrupts.CPU110.PMI:Performance_monitoring_interrupts
      9725           +14.6%      11140        interrupts.CPU111.NMI:Non-maskable_interrupts
      9725           +14.6%      11140        interrupts.CPU111.PMI:Performance_monitoring_interrupts
      9724           +14.5%      11139        interrupts.CPU112.NMI:Non-maskable_interrupts
      9724           +14.5%      11139        interrupts.CPU112.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11139        interrupts.CPU113.NMI:Non-maskable_interrupts
      9723           +14.6%      11139        interrupts.CPU113.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11141        interrupts.CPU114.NMI:Non-maskable_interrupts
      9723           +14.6%      11141        interrupts.CPU114.PMI:Performance_monitoring_interrupts
      9723           +14.6%      11142        interrupts.CPU115.NMI:Non-maskable_interrupts
      9723           +14.6%      11142        interrupts.CPU115.PMI:Performance_monitoring_interrupts
      9725           +14.6%      11141        interrupts.CPU116.NMI:Non-maskable_interrupts
      9725           +14.6%      11141        interrupts.CPU116.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11140        interrupts.CPU118.NMI:Non-maskable_interrupts
      9725           +14.5%      11140        interrupts.CPU118.PMI:Performance_monitoring_interrupts
      9728           +14.5%      11138        interrupts.CPU119.NMI:Non-maskable_interrupts
      9728           +14.5%      11138        interrupts.CPU119.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU120.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU120.PMI:Performance_monitoring_interrupts
      9887           +13.2%      11192        interrupts.CPU121.NMI:Non-maskable_interrupts
      9887           +13.2%      11192        interrupts.CPU121.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11196        interrupts.CPU122.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11196        interrupts.CPU122.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11195        interrupts.CPU123.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11195        interrupts.CPU123.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU124.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU124.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU125.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11197        interrupts.CPU125.PMI:Performance_monitoring_interrupts
      8514 ± 24%     +30.8%      11134        interrupts.CPU13.NMI:Non-maskable_interrupts
      8514 ± 24%     +30.8%      11134        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      9887           +13.2%      11196        interrupts.CPU132.NMI:Non-maskable_interrupts
      9887           +13.2%      11196        interrupts.CPU132.PMI:Performance_monitoring_interrupts
     18958 ±153%     -98.1%     355.75 ± 47%  interrupts.CPU132.RES:Rescheduling_interrupts
      9886           +13.3%      11197        interrupts.CPU133.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU133.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11197        interrupts.CPU134.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU134.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11194        interrupts.CPU135.NMI:Non-maskable_interrupts
      9886           +13.2%      11194        interrupts.CPU135.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11196        interrupts.CPU136.NMI:Non-maskable_interrupts
      9886           +13.2%      11196        interrupts.CPU136.PMI:Performance_monitoring_interrupts
      9884           +13.3%      11197        interrupts.CPU137.NMI:Non-maskable_interrupts
      9884           +13.3%      11197        interrupts.CPU137.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11197        interrupts.CPU138.NMI:Non-maskable_interrupts
      9886           +13.3%      11197        interrupts.CPU138.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11195        interrupts.CPU139.NMI:Non-maskable_interrupts
      9885           +13.3%      11195        interrupts.CPU139.PMI:Performance_monitoring_interrupts
      3747 ±142%     -91.7%     312.50 ± 41%  interrupts.CPU139.RES:Rescheduling_interrupts
      9729           +14.4%      11133        interrupts.CPU14.NMI:Non-maskable_interrupts
      9729           +14.4%      11133        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11196        interrupts.CPU140.NMI:Non-maskable_interrupts
      9885           +13.3%      11196        interrupts.CPU140.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11196        interrupts.CPU141.NMI:Non-maskable_interrupts
      9886           +13.3%      11196        interrupts.CPU141.PMI:Performance_monitoring_interrupts
      8651 ± 24%     +29.4%      11198        interrupts.CPU142.NMI:Non-maskable_interrupts
      8651 ± 24%     +29.4%      11198        interrupts.CPU142.PMI:Performance_monitoring_interrupts
      7413 ± 33%     +51.0%      11194        interrupts.CPU143.NMI:Non-maskable_interrupts
      7413 ± 33%     +51.0%      11194        interrupts.CPU143.PMI:Performance_monitoring_interrupts
      5609 ±160%     -95.0%     279.75 ± 26%  interrupts.CPU143.RES:Rescheduling_interrupts
      8737 ± 24%     +28.1%      11194        interrupts.CPU144.NMI:Non-maskable_interrupts
      8737 ± 24%     +28.1%      11194        interrupts.CPU144.PMI:Performance_monitoring_interrupts
    712.50 ± 58%     -64.7%     251.25 ± 35%  interrupts.CPU144.RES:Rescheduling_interrupts
      7490 ± 33%     +49.5%      11195        interrupts.CPU145.NMI:Non-maskable_interrupts
      7490 ± 33%     +49.5%      11195        interrupts.CPU145.PMI:Performance_monitoring_interrupts
    672.00 ± 49%     -57.4%     286.00 ± 52%  interrupts.CPU145.RES:Rescheduling_interrupts
    501.75 ± 11%     -52.3%     239.50 ± 36%  interrupts.CPU146.RES:Rescheduling_interrupts
    474.25 ± 23%     -43.4%     268.25 ± 38%  interrupts.CPU147.RES:Rescheduling_interrupts
    455.25 ± 22%     -53.7%     210.75 ± 29%  interrupts.CPU149.RES:Rescheduling_interrupts
      9732           +14.4%      11135        interrupts.CPU15.NMI:Non-maskable_interrupts
      9732           +14.4%      11135        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      6244 ± 34%     +56.9%       9796 ± 24%  interrupts.CPU150.NMI:Non-maskable_interrupts
      6244 ± 34%     +56.9%       9796 ± 24%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
    649.75 ± 36%     -65.6%     223.50 ± 25%  interrupts.CPU150.RES:Rescheduling_interrupts
      6243 ± 34%     +34.5%       8395 ± 33%  interrupts.CPU151.NMI:Non-maskable_interrupts
      6243 ± 34%     +34.5%       8395 ± 33%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
    722.50 ± 33%     -72.4%     199.50 ± 38%  interrupts.CPU151.RES:Rescheduling_interrupts
    457.75 ± 20%     -57.2%     195.75 ± 50%  interrupts.CPU152.RES:Rescheduling_interrupts
    816.00 ± 52%     -79.2%     169.75 ± 41%  interrupts.CPU153.RES:Rescheduling_interrupts
    682.50 ± 59%     -77.5%     153.25 ± 36%  interrupts.CPU154.RES:Rescheduling_interrupts
    453.75 ± 19%     -66.4%     152.25 ± 42%  interrupts.CPU155.RES:Rescheduling_interrupts
    847.00 ± 70%     -80.9%     161.75 ± 30%  interrupts.CPU156.RES:Rescheduling_interrupts
      7489 ± 33%     +49.5%      11197        interrupts.CPU157.NMI:Non-maskable_interrupts
      7489 ± 33%     +49.5%      11197        interrupts.CPU157.PMI:Performance_monitoring_interrupts
    506.75 ± 27%     -69.4%     155.00 ± 87%  interrupts.CPU157.RES:Rescheduling_interrupts
      7488 ± 33%     +49.5%      11196        interrupts.CPU158.NMI:Non-maskable_interrupts
      7488 ± 33%     +49.5%      11196        interrupts.CPU158.PMI:Performance_monitoring_interrupts
    691.50 ± 69%     -80.6%     134.00 ± 75%  interrupts.CPU158.RES:Rescheduling_interrupts
      7490 ± 33%     +49.5%      11198        interrupts.CPU159.NMI:Non-maskable_interrupts
      7490 ± 33%     +49.5%      11198        interrupts.CPU159.PMI:Performance_monitoring_interrupts
    753.00 ± 72%     -74.7%     190.25 ± 79%  interrupts.CPU159.RES:Rescheduling_interrupts
      9731           +14.4%      11133        interrupts.CPU16.NMI:Non-maskable_interrupts
      9731           +14.4%      11133        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      8738 ± 24%     +28.2%      11198        interrupts.CPU160.NMI:Non-maskable_interrupts
      8738 ± 24%     +28.2%      11198        interrupts.CPU160.PMI:Performance_monitoring_interrupts
    717.00 ± 65%     -80.0%     143.50 ± 39%  interrupts.CPU160.RES:Rescheduling_interrupts
      6239 ± 34%     +79.5%      11198        interrupts.CPU161.NMI:Non-maskable_interrupts
      6239 ± 34%     +79.5%      11198        interrupts.CPU161.PMI:Performance_monitoring_interrupts
    481.25 ± 25%     -77.5%     108.50 ± 56%  interrupts.CPU161.RES:Rescheduling_interrupts
      6238 ± 34%     +79.5%      11198        interrupts.CPU162.NMI:Non-maskable_interrupts
      6238 ± 34%     +79.5%      11198        interrupts.CPU162.PMI:Performance_monitoring_interrupts
      4993          +124.2%      11196        interrupts.CPU163.NMI:Non-maskable_interrupts
      4993          +124.2%      11196        interrupts.CPU163.PMI:Performance_monitoring_interrupts
    701.75 ± 60%     -82.9%     120.25 ± 67%  interrupts.CPU163.RES:Rescheduling_interrupts
      6245 ± 34%     +79.3%      11196        interrupts.CPU164.NMI:Non-maskable_interrupts
      6245 ± 34%     +79.3%      11196        interrupts.CPU164.PMI:Performance_monitoring_interrupts
    465.00 ± 21%     -65.6%     159.75 ± 45%  interrupts.CPU164.RES:Rescheduling_interrupts
      6244 ± 34%     +79.3%      11195        interrupts.CPU165.NMI:Non-maskable_interrupts
      6244 ± 34%     +79.3%      11195        interrupts.CPU165.PMI:Performance_monitoring_interrupts
    503.50 ± 23%     -75.0%     126.00 ± 86%  interrupts.CPU165.RES:Rescheduling_interrupts
      6244 ± 34%     +79.3%      11194        interrupts.CPU166.NMI:Non-maskable_interrupts
      6244 ± 34%     +79.3%      11194        interrupts.CPU166.PMI:Performance_monitoring_interrupts
    571.25 ± 29%     -76.5%     134.50 ± 76%  interrupts.CPU166.RES:Rescheduling_interrupts
      6243 ± 34%     +56.9%       9794 ± 24%  interrupts.CPU167.NMI:Non-maskable_interrupts
      6243 ± 34%     +56.9%       9794 ± 24%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
    759.75 ± 70%     -84.1%     121.00 ± 45%  interrupts.CPU167.RES:Rescheduling_interrupts
      5615 ±  3%      +5.5%       5924 ±  5%  interrupts.CPU168.CAL:Function_call_interrupts
      9728           +14.5%      11135        interrupts.CPU17.NMI:Non-maskable_interrupts
      9728           +14.5%      11135        interrupts.CPU17.PMI:Performance_monitoring_interrupts
    601.25 ± 41%     -89.1%      65.25 ± 71%  interrupts.CPU170.RES:Rescheduling_interrupts
      5583           +12.7%       6294 ±  8%  interrupts.CPU171.CAL:Function_call_interrupts
    446.00 ± 26%     -91.0%      40.25 ± 28%  interrupts.CPU172.RES:Rescheduling_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU173.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU173.PMI:Performance_monitoring_interrupts
      6032 ± 34%     +61.6%       9745 ± 24%  interrupts.CPU174.NMI:Non-maskable_interrupts
      6032 ± 34%     +61.6%       9745 ± 24%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
      6031 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU175.NMI:Non-maskable_interrupts
      6031 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
      2235 ±125%     -97.1%      65.75 ± 65%  interrupts.CPU175.RES:Rescheduling_interrupts
      6032 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU176.NMI:Non-maskable_interrupts
      6032 ± 34%     +61.5%       9742 ± 24%  interrupts.CPU176.PMI:Performance_monitoring_interrupts
    444.50 ± 33%     -89.0%      48.75 ± 67%  interrupts.CPU176.RES:Rescheduling_interrupts
      6031 ± 34%     +84.7%      11139        interrupts.CPU177.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.7%      11139        interrupts.CPU177.PMI:Performance_monitoring_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU178.NMI:Non-maskable_interrupts
      6031 ± 34%     +84.6%      11136        interrupts.CPU178.PMI:Performance_monitoring_interrupts
      5559 ±  3%      +8.0%       6004 ±  5%  interrupts.CPU179.CAL:Function_call_interrupts
      4829          +130.7%      11139        interrupts.CPU179.NMI:Non-maskable_interrupts
      4829          +130.7%      11139        interrupts.CPU179.PMI:Performance_monitoring_interrupts
      9729           +14.5%      11136        interrupts.CPU18.NMI:Non-maskable_interrupts
      9729           +14.5%      11136        interrupts.CPU18.PMI:Performance_monitoring_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU180.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU180.PMI:Performance_monitoring_interrupts
      6041 ± 34%     +84.4%      11139        interrupts.CPU181.NMI:Non-maskable_interrupts
      6041 ± 34%     +84.4%      11139        interrupts.CPU181.PMI:Performance_monitoring_interrupts
      6040 ± 34%     +84.4%      11138        interrupts.CPU182.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11138        interrupts.CPU182.PMI:Performance_monitoring_interrupts
    410.50 ± 30%     -81.1%      77.75 ± 99%  interrupts.CPU182.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11134        interrupts.CPU183.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11134        interrupts.CPU183.PMI:Performance_monitoring_interrupts
    613.25 ± 57%     -95.1%      30.25 ± 82%  interrupts.CPU183.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11136        interrupts.CPU184.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11136        interrupts.CPU184.PMI:Performance_monitoring_interrupts
      5181 ±149%     -96.2%     197.00 ±127%  interrupts.CPU184.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11135        interrupts.CPU185.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11135        interrupts.CPU185.PMI:Performance_monitoring_interrupts
    461.25 ± 43%     -88.1%      55.00 ± 53%  interrupts.CPU185.RES:Rescheduling_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU186.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.4%      11137        interrupts.CPU186.PMI:Performance_monitoring_interrupts
    800.25 ± 56%     -88.3%      94.00 ± 84%  interrupts.CPU186.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11136        interrupts.CPU187.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11136        interrupts.CPU187.PMI:Performance_monitoring_interrupts
    808.00 ± 53%     -77.1%     185.25 ±132%  interrupts.CPU187.RES:Rescheduling_interrupts
      6040 ± 34%     +84.5%      11144        interrupts.CPU188.NMI:Non-maskable_interrupts
      6040 ± 34%     +84.5%      11144        interrupts.CPU188.PMI:Performance_monitoring_interrupts
    769.00 ± 51%     -79.7%     156.00 ±143%  interrupts.CPU188.RES:Rescheduling_interrupts
      6039 ± 34%     +84.4%      11138        interrupts.CPU189.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11138        interrupts.CPU189.PMI:Performance_monitoring_interrupts
    851.50 ± 45%     -81.6%     156.50 ±126%  interrupts.CPU189.RES:Rescheduling_interrupts
      9728           +14.5%      11135        interrupts.CPU19.NMI:Non-maskable_interrupts
      9728           +14.5%      11135        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      6039 ± 34%     +84.4%      11137        interrupts.CPU190.NMI:Non-maskable_interrupts
      6039 ± 34%     +84.4%      11137        interrupts.CPU190.PMI:Performance_monitoring_interrupts
    746.00 ± 39%     -94.9%      37.75 ± 73%  interrupts.CPU190.RES:Rescheduling_interrupts
      7241 ± 33%     +53.7%      11132        interrupts.CPU191.NMI:Non-maskable_interrupts
      7241 ± 33%     +53.7%      11132        interrupts.CPU191.PMI:Performance_monitoring_interrupts
      9730           +14.4%      11135        interrupts.CPU2.NMI:Non-maskable_interrupts
      9730           +14.4%      11135        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11133        interrupts.CPU20.NMI:Non-maskable_interrupts
      9729           +14.4%      11133        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11135        interrupts.CPU21.NMI:Non-maskable_interrupts
      9729           +14.4%      11135        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      9730           +14.4%      11133        interrupts.CPU22.NMI:Non-maskable_interrupts
      9730           +14.4%      11133        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11134        interrupts.CPU23.NMI:Non-maskable_interrupts
      9727           +14.5%      11134        interrupts.CPU23.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU24.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU24.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU25.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11193        interrupts.CPU26.NMI:Non-maskable_interrupts
      9884           +13.2%      11193        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    388.25 ± 25%    +635.5%       2855 ±139%  interrupts.CPU26.RES:Rescheduling_interrupts
      9886           +13.2%      11191        interrupts.CPU27.NMI:Non-maskable_interrupts
      9886           +13.2%      11191        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11195        interrupts.CPU28.NMI:Non-maskable_interrupts
      9885           +13.3%      11195        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11194        interrupts.CPU29.NMI:Non-maskable_interrupts
      9885           +13.2%      11194        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      9730           +14.5%      11138        interrupts.CPU3.NMI:Non-maskable_interrupts
      9730           +14.5%      11138        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      5512 ±  2%     +29.1%       7118 ± 30%  interrupts.CPU30.CAL:Function_call_interrupts
    396.00 ± 23%     +64.1%     650.00 ± 31%  interrupts.CPU30.RES:Rescheduling_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU31.NMI:Non-maskable_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11196        interrupts.CPU32.NMI:Non-maskable_interrupts
      9885           +13.3%      11196        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11198        interrupts.CPU33.NMI:Non-maskable_interrupts
      9886           +13.3%      11198        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      9885           +13.3%      11197        interrupts.CPU34.NMI:Non-maskable_interrupts
      9885           +13.3%      11197        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11195        interrupts.CPU35.NMI:Non-maskable_interrupts
      9885           +13.2%      11195        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      9886           +13.2%      11195        interrupts.CPU36.NMI:Non-maskable_interrupts
      9886           +13.2%      11195        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      9886           +13.3%      11196        interrupts.CPU37.NMI:Non-maskable_interrupts
      9886           +13.3%      11196        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU38.NMI:Non-maskable_interrupts
      8648 ± 24%     +29.5%      11196        interrupts.CPU38.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11190        interrupts.CPU39.NMI:Non-maskable_interrupts
      9885           +13.2%      11190        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      9729           +14.5%      11137        interrupts.CPU4.NMI:Non-maskable_interrupts
      9729           +14.5%      11137        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU40.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU40.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11193        interrupts.CPU41.NMI:Non-maskable_interrupts
      9884           +13.2%      11193        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11194        interrupts.CPU42.NMI:Non-maskable_interrupts
      9885           +13.2%      11194        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      9884           +13.2%      11190        interrupts.CPU43.NMI:Non-maskable_interrupts
      9884           +13.2%      11190        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      9885           +13.2%      11193        interrupts.CPU44.NMI:Non-maskable_interrupts
      9885           +13.2%      11193        interrupts.CPU44.PMI:Performance_monitoring_interrupts
      9884           +13.3%      11194        interrupts.CPU45.NMI:Non-maskable_interrupts
      9884           +13.3%      11194        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      9883           +13.3%      11194        interrupts.CPU46.NMI:Non-maskable_interrupts
      9883           +13.3%      11194        interrupts.CPU46.PMI:Performance_monitoring_interrupts
      9880           +13.3%      11194        interrupts.CPU47.NMI:Non-maskable_interrupts
      9880           +13.3%      11194        interrupts.CPU47.PMI:Performance_monitoring_interrupts
      9997           +12.0%      11196        interrupts.CPU48.NMI:Non-maskable_interrupts
      9997           +12.0%      11196        interrupts.CPU48.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11194        interrupts.CPU49.NMI:Non-maskable_interrupts
      9987           +12.1%      11194        interrupts.CPU49.PMI:Performance_monitoring_interrupts
     10211 ±156%     -96.1%     402.50 ± 25%  interrupts.CPU49.RES:Rescheduling_interrupts
      9730           +14.4%      11136        interrupts.CPU5.NMI:Non-maskable_interrupts
      9730           +14.4%      11136        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11195        interrupts.CPU50.NMI:Non-maskable_interrupts
      9987           +12.1%      11195        interrupts.CPU50.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11195        interrupts.CPU51.NMI:Non-maskable_interrupts
      9989           +12.1%      11195        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      9986           +12.1%      11194        interrupts.CPU52.NMI:Non-maskable_interrupts
      9986           +12.1%      11194        interrupts.CPU52.PMI:Performance_monitoring_interrupts
    848.00 ± 42%     -49.0%     432.50 ± 38%  interrupts.CPU52.RES:Rescheduling_interrupts
      9990           +12.1%      11198        interrupts.CPU53.NMI:Non-maskable_interrupts
      9990           +12.1%      11198        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11196        interrupts.CPU54.NMI:Non-maskable_interrupts
      9989           +12.1%      11196        interrupts.CPU54.PMI:Performance_monitoring_interrupts
    651.25 ± 34%     -35.5%     420.00 ± 23%  interrupts.CPU54.RES:Rescheduling_interrupts
      9990           +12.1%      11196        interrupts.CPU55.NMI:Non-maskable_interrupts
      9990           +12.1%      11196        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11196        interrupts.CPU56.NMI:Non-maskable_interrupts
      9987           +12.1%      11196        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11195        interrupts.CPU57.NMI:Non-maskable_interrupts
      9989           +12.1%      11195        interrupts.CPU57.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11197        interrupts.CPU58.NMI:Non-maskable_interrupts
      9990           +12.1%      11197        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    572.75 ± 34%     -32.4%     387.00 ± 24%  interrupts.CPU58.RES:Rescheduling_interrupts
      9988           +12.1%      11198        interrupts.CPU59.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      9727           +14.5%      11137        interrupts.CPU6.NMI:Non-maskable_interrupts
      9727           +14.5%      11137        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11197        interrupts.CPU60.NMI:Non-maskable_interrupts
      9990           +12.1%      11197        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      2296 ±128%     -80.8%     440.50 ± 28%  interrupts.CPU60.RES:Rescheduling_interrupts
      9988           +12.1%      11198        interrupts.CPU61.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU61.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11197        interrupts.CPU62.NMI:Non-maskable_interrupts
      9987           +12.1%      11197        interrupts.CPU62.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11198        interrupts.CPU63.NMI:Non-maskable_interrupts
      9988           +12.1%      11198        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11199        interrupts.CPU64.NMI:Non-maskable_interrupts
      9989           +12.1%      11199        interrupts.CPU64.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11200        interrupts.CPU65.NMI:Non-maskable_interrupts
      9988           +12.1%      11200        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      9988           +12.1%      11197        interrupts.CPU66.NMI:Non-maskable_interrupts
      9988           +12.1%      11197        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11197        interrupts.CPU67.NMI:Non-maskable_interrupts
      9987           +12.1%      11197        interrupts.CPU67.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11197        interrupts.CPU68.NMI:Non-maskable_interrupts
      9989           +12.1%      11197        interrupts.CPU68.PMI:Performance_monitoring_interrupts
      9990           +12.1%      11199        interrupts.CPU69.NMI:Non-maskable_interrupts
      9990           +12.1%      11199        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5435            +7.2%       5825 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
      9727           +14.6%      11146        interrupts.CPU7.NMI:Non-maskable_interrupts
      9727           +14.6%      11146        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      9989           +12.1%      11197        interrupts.CPU70.NMI:Non-maskable_interrupts
      9989           +12.1%      11197        interrupts.CPU70.PMI:Performance_monitoring_interrupts
      9987           +12.1%      11195        interrupts.CPU71.NMI:Non-maskable_interrupts
      9987           +12.1%      11195        interrupts.CPU71.PMI:Performance_monitoring_interrupts
      9652           +15.4%      11141        interrupts.CPU72.NMI:Non-maskable_interrupts
      9652           +15.4%      11141        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      9654           +15.6%      11160        interrupts.CPU74.NMI:Non-maskable_interrupts
      9654           +15.6%      11160        interrupts.CPU74.PMI:Performance_monitoring_interrupts
      5564           +12.3%       6246 ±  7%  interrupts.CPU75.CAL:Function_call_interrupts
      9654           +15.3%      11133        interrupts.CPU75.NMI:Non-maskable_interrupts
      9654           +15.3%      11133        interrupts.CPU75.PMI:Performance_monitoring_interrupts
      9653           +15.4%      11135        interrupts.CPU76.NMI:Non-maskable_interrupts
      9653           +15.4%      11135        interrupts.CPU76.PMI:Performance_monitoring_interrupts
    447.50 ± 32%     -31.8%     305.00 ±  9%  interrupts.CPU76.RES:Rescheduling_interrupts
      9654           +15.4%      11139        interrupts.CPU78.NMI:Non-maskable_interrupts
      9654           +15.4%      11139        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      9729           +14.4%      11134        interrupts.CPU8.NMI:Non-maskable_interrupts
      9729           +14.4%      11134        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      9652           +15.4%      11137        interrupts.CPU82.NMI:Non-maskable_interrupts
      9652           +15.4%      11137        interrupts.CPU82.PMI:Performance_monitoring_interrupts
      9654           +15.4%      11137        interrupts.CPU85.NMI:Non-maskable_interrupts
      9654           +15.4%      11137        interrupts.CPU85.PMI:Performance_monitoring_interrupts
    537.25 ± 45%     -50.3%     267.25 ± 13%  interrupts.CPU87.RES:Rescheduling_interrupts
      3077 ±132%     -86.5%     416.00 ± 65%  interrupts.CPU88.RES:Rescheduling_interrupts
      9652           +15.4%      11138        interrupts.CPU89.NMI:Non-maskable_interrupts
      9652           +15.4%      11138        interrupts.CPU89.PMI:Performance_monitoring_interrupts
    494.25 ± 48%     -44.9%     272.25 ± 16%  interrupts.CPU89.RES:Rescheduling_interrupts
      9730           +14.5%      11136        interrupts.CPU9.NMI:Non-maskable_interrupts
      9730           +14.5%      11136        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      9654           +15.4%      11141        interrupts.CPU90.NMI:Non-maskable_interrupts
      9654           +15.4%      11141        interrupts.CPU90.PMI:Performance_monitoring_interrupts
      9653           +15.4%      11140        interrupts.CPU91.NMI:Non-maskable_interrupts
      9653           +15.4%      11140        interrupts.CPU91.PMI:Performance_monitoring_interrupts
    707.50 ± 29%     -44.2%     395.00 ± 66%  interrupts.CPU91.RES:Rescheduling_interrupts
      9652           +15.5%      11149        interrupts.CPU92.NMI:Non-maskable_interrupts
      9652           +15.5%      11149        interrupts.CPU92.PMI:Performance_monitoring_interrupts
    666.50 ± 30%     -46.6%     356.00 ± 58%  interrupts.CPU92.RES:Rescheduling_interrupts
      9651           +15.6%      11153        interrupts.CPU93.NMI:Non-maskable_interrupts
      9651           +15.6%      11153        interrupts.CPU93.PMI:Performance_monitoring_interrupts
    744.75 ± 26%     -49.6%     375.00 ± 62%  interrupts.CPU93.RES:Rescheduling_interrupts
      9651           +15.4%      11141        interrupts.CPU94.NMI:Non-maskable_interrupts
      9651           +15.4%      11141        interrupts.CPU94.PMI:Performance_monitoring_interrupts
    760.50 ± 43%     -68.5%     239.25 ± 16%  interrupts.CPU94.RES:Rescheduling_interrupts
      9658           +15.5%      11155        interrupts.CPU95.NMI:Non-maskable_interrupts
      9658           +15.5%      11155        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11140        interrupts.CPU96.NMI:Non-maskable_interrupts
      9725           +14.5%      11140        interrupts.CPU96.PMI:Performance_monitoring_interrupts
      8510 ± 24%     +30.9%      11138        interrupts.CPU97.NMI:Non-maskable_interrupts
      8510 ± 24%     +30.9%      11138        interrupts.CPU97.PMI:Performance_monitoring_interrupts
      9725           +14.5%      11136        interrupts.CPU98.NMI:Non-maskable_interrupts
      9725           +14.5%      11136        interrupts.CPU98.PMI:Performance_monitoring_interrupts
    350.50 ±  3%     +58.9%     557.00 ±  5%  interrupts.IWI:IRQ_work_interrupts
   1715066 ±  3%     +21.3%    2079884 ±  3%  interrupts.NMI:Non-maskable_interrupts
   1715066 ±  3%     +21.3%    2079884 ±  3%  interrupts.PMI:Performance_monitoring_interrupts
    532498 ± 17%     -67.4%     173726 ± 13%  interrupts.RES:Rescheduling_interrupts

will-it-scale.per_thread_ops

400000 +------------------------------------------------------------------+

         |.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.|   
  350000 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  300000 |-+                                                                |   
         |                                                                  |   
  250000 |-+                                                                |   
         |                                                                  |   
  200000 |-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  150000 |-+       O  O O O O O O O O O O  O                                |   
         | O O O O                           O O O O O O O O                |   
  100000 +------------------------------------------------------------------+   

will-it-scale.workload

8e+07 +-------------------------------------------------------------------+

        |                                                   .+. .+.. .+. .+.|   
  7e+07 |.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+   +    +   +   |   
        |                                                                   |   
        |                                                                   |   
  6e+07 |-+                                                                 |   
        |                                                                   |   
  5e+07 |-+                                                                 |   
        |                                                                   |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  3e+07 |-+        O O O O O O O  O O O O O                                 |   
        | O O O  O                          O O O  O O O O O                |   
  2e+07 +-------------------------------------------------------------------+   

[*] bisect-good sample
[O] bisect-bad sample

***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-skl-fpga01/context_switch1/will-it-scale/0x2000065

commit:
87e867b426 ("sched/pelt: Cleanup PELT divider")
e9f2fb8893 ("sched_pair_cpu: Introduce scheduler task pairing system call")

87e867b4269f29da e9f2fb8893b705bcabebf3c62b6
---------------- ---------------------------

         %stddev     %change         %stddev
             \          |                \  
    218931           -37.5%     136756        will-it-scale.per_thread_ops
  22768896           -37.5%   14222677        will-it-scale.workload
      3418 ±  2%      -2.3%       3338        boot-time.idle
     22.52 ±  4%      -6.2       16.35        mpstat.cpu.all.usr%
      3559 ± 23%     +42.6%       5074 ± 17%  numa-vmstat.node0.nr_inactive_anon
      3933 ± 24%     +39.8%       5496 ± 11%  numa-vmstat.node0.nr_shmem
      3559 ± 23%     +42.6%       5074 ± 17%  numa-vmstat.node0.nr_zone_inactive_anon
     20298            +8.5%      22033        proc-vmstat.nr_kernel_stack
     23006            +9.1%      25092 ±  3%  proc-vmstat.nr_shmem
     23193           +13.9%      26421 ±  4%  proc-vmstat.pgactivate
     76.25            +7.9%      82.25        vmstat.cpu.sy
     21.75 ±  3%     -27.6%      15.75 ±  2%  vmstat.cpu.us
  22550823           -37.5%   14090838        vmstat.system.cs
     14397 ± 23%     +41.5%      20378 ± 16%  numa-meminfo.node0.Inactive
     14237 ± 23%     +42.6%      20297 ± 17%  numa-meminfo.node0.Inactive(anon)
     15734 ± 24%     +39.7%      21988 ± 11%  numa-meminfo.node0.Shmem
     14514 ± 23%     -41.2%       8541 ± 37%  numa-meminfo.node1.Inactive(anon)
    -38316          +109.2%     -80155        sched_debug.cfs_rq:/.spread0.avg
     56956 ± 57%     -77.7%      12727 ±151%  sched_debug.cfs_rq:/.spread0.max
   -216970           +38.1%    -299693        sched_debug.cfs_rq:/.spread0.min
    311271 ± 16%     +40.3%     436737 ±  9%  sched_debug.cpu.avg_idle.max
      2480 ±  4%     +38.1%       3425 ±  2%  sched_debug.cpu.avg_idle.min
     69363 ±  3%     +12.9%      78335 ±  9%  sched_debug.cpu.avg_idle.stddev
      2783 ± 68%    +176.8%       7705 ± 17%  sched_debug.cpu.max_idle_balance_cost.stddev
  32676933           -37.3%   20482288        sched_debug.cpu.nr_switches.avg
  34658910           -39.9%   20829486        sched_debug.cpu.nr_switches.max
  30785157 ±  2%     -35.0%   20010884        sched_debug.cpu.nr_switches.min
   1055140 ± 18%     -84.8%     159933 ± 19%  sched_debug.cpu.nr_switches.stddev
    782.00 ±  6%     -19.6%     628.50 ± 11%  slabinfo.file_lock_cache.active_objs
    782.00 ±  6%     -19.6%     628.50 ± 11%  slabinfo.file_lock_cache.num_objs
     47986           +77.6%      85231        slabinfo.kmalloc-64.active_objs
    750.50           +77.5%       1332        slabinfo.kmalloc-64.active_slabs
     48043           +77.5%      85274        slabinfo.kmalloc-64.num_objs
    750.50           +77.5%       1332        slabinfo.kmalloc-64.num_slabs
    263.50 ± 10%     +29.4%     341.00 ±  9%  slabinfo.numa_policy.active_objs
    263.50 ± 10%     +29.4%     341.00 ±  9%  slabinfo.numa_policy.num_objs
      2385 ±  3%      +9.0%       2600 ±  2%  slabinfo.sighand_cache.active_objs
      2386 ±  3%      +9.0%       2600 ±  2%  slabinfo.sighand_cache.num_objs
      3693 ±  4%      +8.9%       4021 ±  2%  slabinfo.signal_cache.active_objs
      3693 ±  4%      +8.9%       4021 ±  2%  slabinfo.signal_cache.num_objs
      1469 ±  2%     +10.0%       1616 ±  2%  slabinfo.task_struct.active_objs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.active_slabs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.num_objs
      1473 ±  2%     +10.0%       1621 ±  2%  slabinfo.task_struct.num_slabs
 3.699e+10           -19.1%  2.991e+10        perf-stat.i.branch-instructions
      1.40            -0.1        1.28        perf-stat.i.branch-miss-rate%
 5.127e+08           -26.0%  3.794e+08        perf-stat.i.branch-misses
      0.36 ± 30%      +7.1        7.43 ±  9%  perf-stat.i.cache-miss-rate%
   1649484 ± 31%   +2393.0%   41121274 ±  2%  perf-stat.i.cache-misses
  22684327           -37.5%   14176299        perf-stat.i.context-switches
      1.49           +34.4%       2.01        perf-stat.i.cpi
 2.771e+11            +2.9%  2.851e+11        perf-stat.i.cpu-cycles
    223279 ± 25%     -96.9%       6941        perf-stat.i.cycles-between-cache-misses
      0.08            -0.0        0.07        perf-stat.i.dTLB-load-miss-rate%
  45500729           -37.4%   28505101        perf-stat.i.dTLB-load-misses
 5.367e+10           -26.7%  3.931e+10        perf-stat.i.dTLB-loads
     67056 ±  3%     -20.2%      53500        perf-stat.i.dTLB-store-misses
 3.325e+10           -26.0%   2.46e+10        perf-stat.i.dTLB-stores
     57.74 ±  2%     +10.8       68.52 ±  4%  perf-stat.i.iTLB-load-miss-rate%
  67964076           -42.7%   38909739        perf-stat.i.iTLB-load-misses
  49449215 ±  4%     -63.9%   17843583 ± 14%  perf-stat.i.iTLB-loads
 1.853e+11           -23.5%  1.418e+11        perf-stat.i.instructions
      2817 ±  2%     +32.4%       3729        perf-stat.i.instructions-per-iTLB-miss
      0.67           -25.5%       0.50        perf-stat.i.ipc
      2.66            +2.9%       2.74        perf-stat.i.metric.GHz
      1.28 ±  9%     -59.7%       0.52        perf-stat.i.metric.K/sec
      1197           -24.2%     907.93        perf-stat.i.metric.M/sec
     90.52            -2.0       88.47        perf-stat.i.node-load-miss-rate%
    138585 ±  7%   +4081.5%    5794873        perf-stat.i.node-load-misses
     22237 ±  7%   +3283.6%     752443        perf-stat.i.node-loads
     76.13 ±  5%     +12.6       88.73        perf-stat.i.node-store-miss-rate%
     14722 ± 26%   +8185.9%    1219857        perf-stat.i.node-store-misses
      5757 ±  5%   +2559.4%     153105        perf-stat.i.node-stores
      1.39            -0.1        1.27        perf-stat.overall.branch-miss-rate%
      0.32 ± 35%      +7.1        7.39 ±  9%  perf-stat.overall.cache-miss-rate%
      1.50           +34.5%       2.01        perf-stat.overall.cpi
    187391 ± 33%     -96.3%       6935 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.08            -0.0        0.07        perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  3%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     57.90 ±  2%     +10.8       68.71 ±  4%  perf-stat.overall.iTLB-load-miss-rate%
      2727           +33.6%       3644        perf-stat.overall.instructions-per-iTLB-miss
      0.67           -25.6%       0.50        perf-stat.overall.ipc
     85.88            +2.6       88.50        perf-stat.overall.node-load-miss-rate%
     70.96 ±  5%     +17.9       88.82        perf-stat.overall.node-store-miss-rate%
   2452546           +22.2%    2998056        perf-stat.overall.path-length
 3.687e+10           -19.1%  2.981e+10        perf-stat.ps.branch-instructions
  5.11e+08           -26.0%  3.781e+08        perf-stat.ps.branch-misses
   1643802 ± 31%   +2393.3%   40984510 ±  2%  perf-stat.ps.cache-misses
  22608494           -37.5%   14128772        perf-stat.ps.context-switches
 2.762e+11            +2.9%  2.841e+11        perf-stat.ps.cpu-cycles
  45350504           -37.3%   28413918        perf-stat.ps.dTLB-load-misses
 5.349e+10           -26.7%  3.918e+10        perf-stat.ps.dTLB-loads
     66900 ±  3%     -20.0%      53521        perf-stat.ps.dTLB-store-misses
 3.314e+10           -26.0%  2.452e+10        perf-stat.ps.dTLB-stores
  67733698           -42.7%   38786731        perf-stat.ps.iTLB-load-misses
  49282420 ±  3%     -63.9%   17783730 ± 14%  perf-stat.ps.iTLB-loads
 1.847e+11           -23.5%  1.413e+11        perf-stat.ps.instructions
    138052 ±  7%   +4083.5%    5775416        perf-stat.ps.node-load-misses
     22671 ±  7%   +3210.8%     750607        perf-stat.ps.node-loads
     14669 ± 26%   +8188.4%    1215837        perf-stat.ps.node-store-misses
      5772 ±  5%   +2550.7%     153020        perf-stat.ps.node-stores
 5.584e+13           -23.6%  4.264e+13        perf-stat.total.instructions
    407830 ±  6%     +24.8%     509113 ±  2%  interrupts.CAL:Function_call_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU0.NMI:Non-maskable_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.1%       7219 ±  6%  interrupts.CPU10.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.1%       7219 ±  6%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU100.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
      7059 ±  6%      +4.4%       7366 ±  6%  interrupts.CPU101.NMI:Non-maskable_interrupts
      7059 ±  6%      +4.4%       7366 ±  6%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      7058 ±  6%      +4.3%       7365 ±  6%  interrupts.CPU102.NMI:Non-maskable_interrupts
      7058 ±  6%      +4.3%       7365 ±  6%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU103.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.1%       7220 ±  6%  interrupts.CPU11.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.1%       7220 ±  6%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU12.NMI:Non-maskable_interrupts
      6674 ±  6%      +8.2%       7220 ±  6%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      6670 ±  6%      +8.3%       7221 ±  6%  interrupts.CPU13.NMI:Non-maskable_interrupts
      6670 ±  6%      +8.3%       7221 ±  6%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      6671 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU14.NMI:Non-maskable_interrupts
      6671 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU15.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU16.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU17.NMI:Non-maskable_interrupts
      6672 ±  6%      +8.2%       7219 ±  6%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU2.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     12615 ±150%     -94.4%     712.25 ± 38%  interrupts.CPU2.RES:Rescheduling_interrupts
      6665 ±  6%      +8.3%       7220 ±  6%  interrupts.CPU24.NMI:Non-maskable_interrupts
      6665 ±  6%      +8.3%       7220 ±  6%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      6665 ±  6%      +8.3%       7219 ±  6%  interrupts.CPU25.NMI:Non-maskable_interrupts
      6665 ±  6%      +8.3%       7219 ±  6%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU26.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU27.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU28.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU29.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU30.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU31.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU32.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU33.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7367 ±  6%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU34.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.4%       7373 ±  6%  interrupts.CPU35.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.4%       7373 ±  6%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU36.NMI:Non-maskable_interrupts
      7063 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU37.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.4%       7370 ±  6%  interrupts.CPU38.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.4%       7370 ±  6%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU39.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7222 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7222 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.4%       7369 ±  6%  interrupts.CPU40.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.4%       7369 ±  6%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.4%       7368 ±  6%  interrupts.CPU41.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.4%       7368 ±  6%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU42.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU43.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7368 ±  6%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      6085 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU44.NMI:Non-maskable_interrupts
      6085 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      6087 ± 20%     +21.0%       7368 ±  6%  interrupts.CPU45.NMI:Non-maskable_interrupts
      6087 ± 20%     +21.0%       7368 ±  6%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU46.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU47.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7368 ±  6%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      6086 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU48.NMI:Non-maskable_interrupts
      6086 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      6084 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU49.NMI:Non-maskable_interrupts
      6084 ± 20%     +21.1%       7367 ±  6%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU5.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      6088 ± 20%     +21.0%       7365 ±  6%  interrupts.CPU51.NMI:Non-maskable_interrupts
      6088 ± 20%     +21.0%       7365 ±  6%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU53.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU54.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU55.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU56.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6651 ±  6%      +8.4%       7210 ±  6%  interrupts.CPU57.NMI:Non-maskable_interrupts
      6651 ±  6%      +8.4%       7210 ±  6%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU58.NMI:Non-maskable_interrupts
      6652 ±  6%      +8.4%       7211 ±  6%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU6.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      5722 ± 20%     +26.1%       7213 ±  6%  interrupts.CPU63.NMI:Non-maskable_interrupts
      5722 ± 20%     +26.1%       7213 ±  6%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
     41159 ±147%     -98.7%     543.50 ± 13%  interrupts.CPU66.RES:Rescheduling_interrupts
      6675 ±  6%      +8.3%       7233 ±  6%  interrupts.CPU7.NMI:Non-maskable_interrupts
      6675 ±  6%      +8.3%       7233 ±  6%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      6645 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU75.NMI:Non-maskable_interrupts
      6645 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      6644 ±  6%      +8.5%       7210 ±  6%  interrupts.CPU76.NMI:Non-maskable_interrupts
      6644 ±  6%      +8.5%       7210 ±  6%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      6641 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU77.NMI:Non-maskable_interrupts
      6641 ±  6%      +8.5%       7209 ±  6%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU78.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU79.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      6686 ±  6%      +8.0%       7221 ±  6%  interrupts.CPU8.NMI:Non-maskable_interrupts
      6686 ±  6%      +8.0%       7221 ±  6%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU80.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU81.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.4%       7367 ±  6%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      7056 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU82.NMI:Non-maskable_interrupts
      7056 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU83.NMI:Non-maskable_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU84.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU85.NMI:Non-maskable_interrupts
      7057 ±  5%      +4.4%       7365 ±  6%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU86.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU87.NMI:Non-maskable_interrupts
      7062 ±  5%      +4.3%       7369 ±  6%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU88.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU89.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU9.NMI:Non-maskable_interrupts
      6676 ±  6%      +8.2%       7221 ±  6%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU90.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU91.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU92.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU93.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      7058 ±  5%      +4.4%       7366 ±  6%  interrupts.CPU94.NMI:Non-maskable_interrupts
      7058 ±  5%      +4.4%       7366 ±  6%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      7059 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU95.NMI:Non-maskable_interrupts
      7059 ±  5%      +4.3%       7364 ±  6%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU96.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
      7072 ±  5%      +4.1%       7365 ±  6%  interrupts.CPU97.NMI:Non-maskable_interrupts
      7072 ±  5%      +4.1%       7365 ±  6%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU98.NMI:Non-maskable_interrupts
      7060 ±  5%      +4.3%       7365 ±  6%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU99.NMI:Non-maskable_interrupts
      7061 ±  5%      +4.3%       7366 ±  6%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
    117.75 ± 15%     +52.0%     179.00 ± 10%  interrupts.IWI:IRQ_work_interrupts
   1034638 ± 10%     -35.2%     670935 ± 11%  interrupts.RES:Rescheduling_interrupts
      9.66 ±  4%      -4.0        5.69 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      9.57 ±  4%      -3.9        5.64 ±  2%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      9.34 ±  4%      -3.9        5.42 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      8.19 ±  4%      -3.4        4.80 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
      6.05            -2.5        3.59        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.73            -1.9        2.82        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.66            -1.7        1.97        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.new_sync_read
      3.09 ±  4%      -1.5        1.61        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_read
      3.75 ±  6%      -1.4        2.31 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      3.07            -1.4        1.69        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_read
     44.10            -1.4       42.73        perf-profile.calltrace.cycles-pp.__GI___libc_write
      3.06            -1.3        1.73        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_write
      3.00 ±  6%      -1.2        1.78 ±  4%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      3.02 ±  4%      -1.2        1.85        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__GI___libc_write

1.89 -1.0 0.92 perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read
3.26 ± 5% -0.9 2.38 perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
2.36 -0.8 1.51 perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
1.89 -0.8 1.05 perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
1.69 -0.8 0.90 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
1.71 -0.8 0.92 perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
1.79 -0.8 1.01 perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
3.01 ± 6% -0.7 2.28 perf-profile.calltrace.cycles-pp.switch_fpu_return.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
1.40 ± 2% -0.7 0.75 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
1.03 -0.6 0.39 ± 57% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
1.30 ± 10% -0.6 0.74 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.25 -0.6 0.70 perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read
1.73 -0.6 1.18 perf-profile.calltrace.cycles-pp.__switch_to.__GI___libc_read
1.13 ± 2% -0.5 0.61 perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
1.20 ± 3% -0.5 0.69 ± 2% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.34 ± 2% -0.5 0.83 ± 3% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.23 ± 10% -0.5 0.72 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
1.06 ± 2% -0.5 0.57 perf-profile.calltrace.cycles-pp.sched_clock_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
1.04 -0.5 0.56 perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__schedule.schedule.pipe_read
1.11 ± 2% -0.5 0.64 perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
1.48 ± 2% -0.5 1.01 perf-profile.calltrace.cycles-pp.__switch_to_asm.__GI___libc_read
1.11 ± 2% -0.4 0.67 perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
0.97 ± 3% -0.4 0.55 ± 2% perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
1.08 ± 6% -0.4 0.66 ± 6% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.98 ± 2% -0.4 0.56 perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
1.07 ± 3% -0.4 0.68 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.80 ± 5% -0.2 0.56 ± 5% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
0.78 ± 5% -0.2 0.55 ± 4% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.27 -0.2 1.09 perf-profile.calltrace.cycles-pp.native_write_msr.__GI___libc_read
0.58 -0.0 0.56 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_read.new_sync_read
37.18 +1.5 38.69 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
36.93 +1.6 38.55 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
0.00 +1.7 1.66 perf-profile.calltrace.cycles-pp.ring_buffer_lock_reserve.trace_vbprintk.__trace_bprintk.__schedule.schedule
54.55 +1.7 56.24 perf-profile.calltrace.cycles-pp.__GI___libc_read
0.00 +2.1 2.12 ± 2% perf-profile.calltrace.cycles-pp.format_decode.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule
31.13 +2.6 33.70 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
29.77 +2.9 32.67 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
0.00 +2.9 2.94 ± 2% perf-profile.calltrace.cycles-pp.vbin_printf.trace_vbprintk.__trace_bprintk.__schedule.schedule
0.00 +3.4 3.35 perf-profile.calltrace.cycles-pp.probe_sched_switch.__schedule.schedule.pipe_read.new_sync_read
0.00 +3.4 3.41 perf-profile.calltrace.cycles-pp.probe_sched_wakeup.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
27.33 +3.8 31.13 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
26.62 +4.1 30.69 perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
42.39 +5.8 48.18 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
43.08 +6.1 49.15 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
21.46 ± 2% +6.2 27.62 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
20.61 ± 2% +6.4 27.05 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
0.00 +6.5 6.46 perf-profile.calltrace.cycles-pp.trace_vbprintk.__trace_bprintk.__schedule.schedule.pipe_read
20.38 ± 2% +6.5 26.91 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
0.00 +6.6 6.56 perf-profile.calltrace.cycles-pp.__trace_bprintk.__schedule.schedule.pipe_read.new_sync_read
19.91 ± 2% +6.7 26.63 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
33.81 +8.2 42.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
32.21 +8.9 41.09 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
28.99 +10.2 39.15 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
27.48 +11.0 38.50 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
0.00 +11.3 11.28 perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
0.00 +11.4 11.37 perf-profile.calltrace.cycles-pp.tracing_record_taskinfo.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +12.2 12.22 perf-profile.calltrace.cycles-pp.trace_save_cmdline.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read
0.00 +12.3 12.29 perf-profile.calltrace.cycles-pp.tracing_record_taskinfo_sched_switch.__schedule.schedule.pipe_read.new_sync_read
1.56 +14.3 15.82 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
17.22 ± 2% +15.4 32.65 perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
16.60 ± 2% +15.7 32.32 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
9.66 ± 4% -4.0 5.70 ± 2% perf-profile.children.cycles-pp.ttwu_do_activate
9.58 ± 4% -3.9 5.65 ± 2% perf-profile.children.cycles-pp.activate_task
9.36 ± 4% -3.9 5.43 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair
8.21 ± 4% -3.4 4.82 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
6.82 ± 4% -2.9 3.89 perf-profile.children.cycles-pp.syscall_return_via_sysret
6.14 -2.7 3.42 perf-profile.children.cycles-pp.entry_SYSCALL_64
5.76 ± 5% -2.6 3.20 ± 2% perf-profile.children.cycles-pp.update_load_avg
6.07 -2.5 3.61 perf-profile.children.cycles-pp.select_task_rq_fair
4.88 ± 2% -2.0 2.86 ± 2% perf-profile.children.cycles-pp.update_curr
4.86 -2.0 2.89 perf-profile.children.cycles-pp.select_idle_sibling
3.91 -1.8 2.09 perf-profile.children.cycles-pp.pick_next_task_fair
3.74 -1.6 2.10 perf-profile.children.cycles-pp.reweight_entity
44.51 -1.5 42.99 perf-profile.children.cycles-pp.__GI___libc_write
3.87 ± 6% -1.5 2.37 ± 3% perf-profile.children.cycles-pp.enqueue_entity
3.07 ± 6% -1.3 1.82 ± 4% perf-profile.children.cycles-pp.dequeue_entity
1.92 -1.0 0.94 perf-profile.children.cycles-pp.copy_page_to_iter
2.46 ± 2% -0.9 1.54 ± 2% perf-profile.children.cycles-pp.security_file_permission
2.44 -0.9 1.56 perf-profile.children.cycles-pp.prepare_to_wait_event
2.04 -0.9 1.17 perf-profile.children.cycles-pp.mutex_lock
1.96 ± 7% -0.8 1.13 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
1.77 -0.8 0.93 perf-profile.children.cycles-pp.set_next_entity
1.76 -0.8 0.95 perf-profile.children.cycles-pp.copy_page_from_iter
3.88 ± 8% -0.8 3.08 perf-profile.children.cycles-pp.__prepare_exit_to_usermode
2.37 -0.8 1.60 perf-profile.children.cycles-pp.__switch_to
1.86 -0.8 1.10 perf-profile.children.cycles-pp.sched_clock_cpu
3.03 ± 6% -0.7 2.29 perf-profile.children.cycles-pp.switch_fpu_return
1.85 -0.7 1.11 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.70 ± 6% -0.7 0.96 perf-profile.children.cycles-pp.update_cfs_group
1.45 -0.7 0.73 perf-profile.children.cycles-pp.__update_load_avg_se
1.54 -0.7 0.83 perf-profile.children.cycles-pp.___might_sleep
1.42 ± 2% -0.7 0.76 perf-profile.children.cycles-pp.check_preempt_curr
1.75 -0.7 1.10 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
1.36 -0.6 0.77 ± 2% perf-profile.children.cycles-pp.__calc_delta
1.42 -0.6 0.83 ± 2% perf-profile.children.cycles-pp.common_file_perm
1.62 ± 5% -0.6 1.04 ± 5% perf-profile.children.cycles-pp.__fdget_pos
1.17 ± 2% -0.5 0.62 perf-profile.children.cycles-pp.check_preempt_wakeup
1.55 ± 2% -0.5 1.04 perf-profile.children.cycles-pp.__switch_to_asm
1.16 ± 3% -0.5 0.64 ± 2% perf-profile.children.cycles-pp.fsnotify
0.99 ± 2% -0.5 0.48 perf-profile.children.cycles-pp.__might_sleep
1.51 ± 5% -0.5 1.00 ± 5% perf-profile.children.cycles-pp.__fget_light
0.90 ± 2% -0.5 0.40 perf-profile.children.cycles-pp.__might_fault
1.20 ± 2% -0.5 0.71 ± 2% perf-profile.children.cycles-pp.cpumask_next_wrap
0.95 ± 9% -0.5 0.47 ± 2% perf-profile.children.cycles-pp.touch_atime
1.16 ± 2% -0.5 0.69 perf-profile.children.cycles-pp._find_next_bit
0.82 ± 8% -0.5 0.35 ± 6% perf-profile.children.cycles-pp.current_time
0.90 -0.5 0.43 perf-profile.children.cycles-pp.pick_next_entity
1.69 -0.4 1.25 perf-profile.children.cycles-pp.sched_clock
0.75 ± 7% -0.4 0.32 ± 2% perf-profile.children.cycles-pp.atime_needs_update
1.61 -0.4 1.21 perf-profile.children.cycles-pp.native_sched_clock
1.21 -0.4 0.81 perf-profile.children.cycles-pp.update_rq_clock
1.09 ± 6% -0.4 0.72 ± 7% perf-profile.children.cycles-pp.__fget_files
0.67 ± 2% -0.4 0.32 perf-profile.children.cycles-pp._cond_resched
0.72 ± 11% -0.3 0.40 ± 5% perf-profile.children.cycles-pp.cpuacct_charge
0.78 ± 7% -0.3 0.47 ± 2% perf-profile.children.cycles-pp.update_min_vruntime
0.70 -0.3 0.40 ± 2% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.65 ± 2% -0.3 0.35 ± 2% perf-profile.children.cycles-pp.cpumask_next
0.71 ± 2% -0.3 0.42 ± 3% perf-profile.children.cycles-pp.fsnotify_parent
0.91 -0.3 0.63 ± 2% perf-profile.children.cycles-pp.__x86_retpoline_rax
0.61 ± 2% -0.3 0.33 ± 3% perf-profile.children.cycles-pp.copyout
0.76 -0.3 0.48 ± 2% perf-profile.children.cycles-pp.__pthread_disable_asynccancel
0.58 -0.3 0.31 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.63 ± 2% -0.3 0.37 ± 4% perf-profile.children.cycles-pp.account_entity_dequeue
0.44 -0.2 0.19 ± 3% perf-profile.children.cycles-pp.clear_buddies
0.68 -0.2 0.45 perf-profile.children.cycles-pp.mutex_unlock
0.58 -0.2 0.36 perf-profile.children.cycles-pp._raw_spin_lock_irq
0.53 -0.2 0.31 ± 2% perf-profile.children.cycles-pp.copyin
1.39 -0.2 1.17 perf-profile.children.cycles-pp.native_write_msr
0.41 ± 2% -0.2 0.20 ± 2% perf-profile.children.cycles-pp.rcu_all_qs
0.58 -0.2 0.38 ± 2% perf-profile.children.cycles-pp.account_entity_enqueue
0.33 ± 15% -0.2 0.12 ± 6% perf-profile.children.cycles-pp.timestamp_truncate
0.51 ± 2% -0.2 0.31 ± 3% perf-profile.children.cycles-pp.available_idle_cpu
0.36 ± 11% -0.2 0.17 ± 6% perf-profile.children.cycles-pp.__sb_start_write
0.54 -0.2 0.36 ± 3% perf-profile.children.cycles-pp._raw_spin_lock
0.42 ± 3% -0.2 0.25 ± 3% perf-profile.children.cycles-pp.finish_wait
0.64 ± 2% -0.2 0.47 perf-profile.children.cycles-pp.___perf_sw_event
0.35 ± 5% -0.2 0.18 ± 2% perf-profile.children.cycles-pp.__syscall_return_slowpath
0.44 ± 2% -0.2 0.28 ± 3% perf-profile.children.cycles-pp.apparmor_file_permission
0.40 ± 15% -0.1 0.25 ± 9% perf-profile.children.cycles-pp.set_next_buddy
0.69 ± 14% -0.1 0.55 ± 2% perf-profile.children.cycles-pp.file_update_time
0.33 -0.1 0.18 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.29 -0.1 0.15 ± 5% perf-profile.children.cycles-pp.__x64_sys_read
0.28 -0.1 0.15 ± 2% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.22 ± 11% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.__sb_end_write
0.62 ± 2% -0.1 0.52 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.51 ± 2% -0.1 0.41 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.50 ± 2% -0.1 0.40 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt
0.22 ± 13% -0.1 0.12 ± 12% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
0.28 ± 2% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.__x64_sys_write
0.56 ± 2% -0.1 0.46 ± 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.26 -0.1 0.16 ± 4% perf-profile.children.cycles-pp.iov_iter_init
0.24 ± 4% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.put_prev_entity
0.17 ± 2% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.rb_erase
0.18 ± 8% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.49 -0.1 0.40 ± 2% perf-profile.children.cycles-pp.fput_many
0.17 ± 6% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.place_entity
0.19 ± 2% -0.1 0.10 perf-profile.children.cycles-pp.__list_add_valid
0.18 ± 2% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.find_next_bit
0.14 -0.1 0.05 ± 9% perf-profile.children.cycles-pp.__enqueue_entity
0.22 ± 5% -0.1 0.14 ± 6% perf-profile.children.cycles-pp.__list_del_entry_valid
0.20 ± 3% -0.1 0.12 ± 5% perf-profile.children.cycles-pp.aa_file_perm
0.15 ± 5% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.check_cfs_rq_runtime
0.11 ± 4% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.rcu_note_context_switch
0.11 ± 3% -0.1 0.05 perf-profile.children.cycles-pp.__bitmap_and
0.21 ± 2% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.tick_sched_handle
0.19 ± 3% -0.1 0.13 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.24 -0.1 0.18 ± 11% perf-profile.children.cycles-pp.tick_sched_timer
0.29 ± 2% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.21 ± 2% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.update_process_times
0.45 ± 2% -0.1 0.39 ± 4% perf-profile.children.cycles-pp.asm_call_on_stack
0.16 ± 4% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.clockevents_program_event
0.16 ± 2% -0.1 0.11 perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.08 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.rw_verify_area
0.14 ± 5% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.ktime_get
0.53 -0.0 0.49 perf-profile.children.cycles-pp.__pthread_enable_asynccancel
0.09 -0.0 0.05 ± 8% perf-profile.children.cycles-pp.default_wake_function
0.20 ± 4% -0.0 0.17 perf-profile.children.cycles-pp.deactivate_task
0.12 ± 4% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.fput
0.12 ± 7% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.scheduler_tick
0.08 ± 8% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.__vfs_write
0.07 ± 5% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.rb_next
0.00 +0.1 0.05 perf-profile.children.cycles-pp.ring_buffer_event_data
0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.hrtick_update
0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.perf_exclude_event
0.00 +0.1 0.09 perf-profile.children.cycles-pp.ring_buffer_nest_end
0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.tracing_record_taskinfo_skip
0.00 +0.1 0.14 ± 3% perf-profile.children.cycles-pp.tracing_generic_entry_update
0.00 +0.2 0.20 ± 4% perf-profile.children.cycles-pp.ring_buffer_nest_start
0.00 +0.2 0.21 ± 2% perf-profile.children.cycles-pp.rb_commit
0.00 +0.3 0.27 perf-profile.children.cycles-pp.__rb_reserve_next
0.00 +0.3 0.32 perf-profile.children.cycles-pp.trace_clock_local
0.00 +0.3 0.34 perf-profile.children.cycles-pp.ring_buffer_unlock_commit
54.94 +1.5 56.47 perf-profile.children.cycles-pp.__GI___libc_read
0.00 +1.7 1.66 perf-profile.children.cycles-pp.ring_buffer_lock_reserve
0.00 +2.2 2.22 ± 2% perf-profile.children.cycles-pp.format_decode
31.17 +2.6 33.74 perf-profile.children.cycles-pp.ksys_write
29.84 +2.9 32.72 perf-profile.children.cycles-pp.vfs_write
0.00 +3.0 3.02 ± 2% perf-profile.children.cycles-pp.vbin_printf
0.00 +3.4 3.37 perf-profile.children.cycles-pp.probe_sched_switch
0.00 +3.4 3.41 perf-profile.children.cycles-pp.probe_sched_wakeup
27.37 +3.8 31.18 perf-profile.children.cycles-pp.new_sync_write
26.69 +4.0 30.74 perf-profile.children.cycles-pp.pipe_write
21.49 ± 2% +6.1 27.64 perf-profile.children.cycles-pp.__wake_up_common_lock
20.61 ± 2% +6.4 27.05 perf-profile.children.cycles-pp.__wake_up_common
20.41 ± 2% +6.5 26.94 perf-profile.children.cycles-pp.autoremove_wake_function
0.00 +6.5 6.54 perf-profile.children.cycles-pp.trace_vbprintk
0.00 +6.6 6.59 perf-profile.children.cycles-pp.__trace_bprintk
19.93 ± 2% +6.7 26.64 perf-profile.children.cycles-pp.try_to_wake_up
79.41 +7.4 86.80 perf-profile.children.cycles-pp.do_syscall_64
80.32 +7.6 87.91 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
33.89 +8.2 42.05 perf-profile.children.cycles-pp.ksys_read
32.29 +8.9 41.15 perf-profile.children.cycles-pp.vfs_read
29.04 +10.1 39.18 perf-profile.children.cycles-pp.new_sync_read
27.63 +11.0 38.59 perf-profile.children.cycles-pp.pipe_read
0.00 +11.4 11.39 perf-profile.children.cycles-pp.tracing_record_taskinfo
0.00 +12.3 12.35 perf-profile.children.cycles-pp.tracing_record_taskinfo_sched_switch
1.58 ± 2% +14.2 15.83 perf-profile.children.cycles-pp.ttwu_do_wakeup
17.33 +15.4 32.74 perf-profile.children.cycles-pp.schedule
16.84 ± 2% +15.7 32.50 perf-profile.children.cycles-pp.__schedule
0.00 +23.6 23.55 perf-profile.children.cycles-pp.trace_save_cmdline
6.81 ± 4% -2.9 3.88 perf-profile.self.cycles-pp.syscall_return_via_sysret
5.44 -2.4 3.00 perf-profile.self.cycles-pp.entry_SYSCALL_64
9.40 ± 2% -2.1 7.32 perf-profile.self.cycles-pp.do_syscall_64
2.39 ± 6% -1.0 1.38 ± 4% perf-profile.self.cycles-pp.update_load_avg
1.89 ± 8% -0.8 1.08 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
2.06 ± 9% -0.8 1.27 ± 7% perf-profile.self.cycles-pp.update_curr
3.01 ± 6% -0.8 2.24 perf-profile.self.cycles-pp.switch_fpu_return
1.67 ± 6% -0.7 0.94 perf-profile.self.cycles-pp.update_cfs_group
2.23 -0.7 1.52 perf-profile.self.cycles-pp.__switch_to
1.78 -0.7 1.07 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.42 -0.7 0.72 perf-profile.self.cycles-pp.__update_load_avg_se
1.51 -0.7 0.82 perf-profile.self.cycles-pp.___might_sleep
1.14 ± 2% -0.7 0.46 ± 2% perf-profile.self.cycles-pp.new_sync_read
1.53 -0.6 0.88 perf-profile.self.cycles-pp.__schedule
1.57 -0.6 0.93 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
1.45 -0.6 0.85 perf-profile.self.cycles-pp.pipe_read
1.35 -0.6 0.76 ± 3% perf-profile.self.cycles-pp.__calc_delta
1.50 ± 5% -0.6 0.94 perf-profile.self.cycles-pp.enqueue_task_fair
1.19 -0.5 0.65 perf-profile.self.cycles-pp.reweight_entity
1.55 ± 2% -0.5 1.03 perf-profile.self.cycles-pp.__switch_to_asm
1.12 ± 3% -0.5 0.62 ± 2% perf-profile.self.cycles-pp.fsnotify
1.22 ± 2% -0.5 0.72 ± 3% perf-profile.self.cycles-pp.common_file_perm
1.21 -0.5 0.72 perf-profile.self.cycles-pp.select_task_rq_fair
1.42 -0.5 0.93 ± 3% perf-profile.self.cycles-pp.select_idle_sibling
1.15 ± 2% -0.5 0.69 perf-profile.self.cycles-pp._find_next_bit
0.89 ± 2% -0.5 0.43 perf-profile.self.cycles-pp.__might_sleep
1.58 -0.4 1.17 perf-profile.self.cycles-pp.native_sched_clock
0.98 -0.4 0.61 perf-profile.self.cycles-pp.dequeue_task_fair
0.78 -0.4 0.41 perf-profile.self.cycles-pp.try_to_wake_up
0.73 ± 3% -0.4 0.36 perf-profile.self.cycles-pp.check_preempt_wakeup
1.05 -0.4 0.69 perf-profile.self.cycles-pp.prepare_to_wait_event
1.06 ± 7% -0.4 0.71 ± 7% perf-profile.self.cycles-pp.__fget_files
0.88 -0.3 0.54 perf-profile.self.cycles-pp.pipe_write
0.55 -0.3 0.21 ± 3% perf-profile.self.cycles-pp.set_next_entity
0.72 ± 11% -0.3 0.40 ± 4% perf-profile.self.cycles-pp.cpuacct_charge
0.83 -0.3 0.52 ± 2% perf-profile.self.cycles-pp.vfs_read
0.76 ± 7% -0.3 0.45 ± 2% perf-profile.self.cycles-pp.update_min_vruntime
0.77 ± 14% -0.3 0.48 ± 9% perf-profile.self.cycles-pp.enqueue_entity
0.66 ± 2% -0.3 0.38 perf-profile.self.cycles-pp.pick_next_task_fair
0.62 -0.3 0.34 perf-profile.self.cycles-pp.__GI___libc_write
0.66 -0.3 0.38 ± 2% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.60 ± 2% -0.3 0.32 ± 2% perf-profile.self.cycles-pp.__GI___libc_read
0.64 -0.3 0.37 perf-profile.self.cycles-pp.fsnotify_parent
0.47 ± 2% -0.3 0.21 ± 2% perf-profile.self.cycles-pp.copy_page_to_iter
0.69 -0.3 0.44 ± 2% perf-profile.self.cycles-pp.__pthread_disable_asynccancel
0.54 -0.3 0.29 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.55 ± 2% -0.2 0.30 perf-profile.self.cycles-pp.ksys_read
0.79 ± 2% -0.2 0.55 ± 2% perf-profile.self.cycles-pp.__x86_retpoline_rax
0.76 -0.2 0.52 perf-profile.self.cycles-pp.mutex_lock
0.47 -0.2 0.24 ± 2% perf-profile.self.cycles-pp.schedule
0.39 -0.2 0.16 ± 2% perf-profile.self.cycles-pp.clear_buddies
1.39 -0.2 1.17 perf-profile.self.cycles-pp.native_write_msr
0.66 -0.2 0.44 perf-profile.self.cycles-pp.mutex_unlock
0.45 -0.2 0.23 perf-profile.self.cycles-pp.pick_next_entity
0.64 -0.2 0.43 ± 5% perf-profile.self.cycles-pp.vfs_write
0.53 ± 2% -0.2 0.32 ± 3% perf-profile.self.cycles-pp.account_entity_dequeue
0.56 -0.2 0.35 perf-profile.self.cycles-pp._raw_spin_lock_irq
0.48 -0.2 0.27 ± 2% perf-profile.self.cycles-pp.cpumask_next_wrap
0.43 -0.2 0.23 perf-profile.self.cycles-pp.copy_page_from_iter
0.35 ± 10% -0.2 0.16 ± 5% perf-profile.self.cycles-pp.__sb_start_write
0.31 ± 16% -0.2 0.12 ± 10% perf-profile.self.cycles-pp.timestamp_truncate
0.53 -0.2 0.35 ± 2% perf-profile.self.cycles-pp.account_entity_enqueue
0.34 -0.2 0.16 perf-profile.self.cycles-pp.rcu_all_qs
0.45 ± 4% -0.2 0.28 ± 4% perf-profile.self.cycles-pp.available_idle_cpu
0.52 -0.2 0.35 ± 3% perf-profile.self.cycles-pp._raw_spin_lock
0.28 ± 4% -0.2 0.11 ± 7% perf-profile.self.cycles-pp.current_time
0.33 ± 6% -0.2 0.17 ± 2% perf-profile.self.cycles-pp.__syscall_return_slowpath
0.28 ± 3% -0.2 0.12 perf-profile.self.cycles-pp._cond_resched
0.57 ± 3% -0.2 0.42 perf-profile.self.cycles-pp.___perf_sw_event
0.49 ± 4% -0.1 0.34 ± 2% perf-profile.self.cycles-pp.new_sync_write
0.39 ± 15% -0.1 0.25 ± 9% perf-profile.self.cycles-pp.set_next_buddy
0.32 ± 2% -0.1 0.18 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.39 ± 3% -0.1 0.26 ± 3% perf-profile.self.cycles-pp.apparmor_file_permission
0.41 -0.1 0.28 perf-profile.self.cycles-pp.update_rq_clock
0.33 ± 2% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.autoremove_wake_function
0.29 ± 9% -0.1 0.17 ± 4% perf-profile.self.cycles-pp.atime_needs_update
0.27 ± 3% -0.1 0.15 ± 5% perf-profile.self.cycles-pp.dequeue_entity
0.26 -0.1 0.14 ± 5% perf-profile.self.cycles-pp.__x64_sys_read
0.38 ± 11% -0.1 0.27 ± 10% perf-profile.self.cycles-pp.__fget_light
0.21 ± 11% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.__sb_end_write
0.26 ± 3% -0.1 0.15 perf-profile.self.cycles-pp.wakeup_preempt_entity
0.21 ± 11% -0.1 0.11 ± 15% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.15 ± 3% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.__fdget_pos
0.23 -0.1 0.13 ± 5% perf-profile.self.cycles-pp.iov_iter_init
0.26 -0.1 0.16 ± 4% perf-profile.self.cycles-pp.ksys_write
0.19 ± 4% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.check_preempt_curr
0.12 -0.1 0.03 ±100% perf-profile.self.cycles-pp.__enqueue_entity
0.17 ± 8% -0.1 0.08 perf-profile.self.cycles-pp.switch_mm_irqs_off
0.15 ± 3% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.rb_erase
0.25 -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__x64_sys_write
0.47 -0.1 0.39 ± 2% perf-profile.self.cycles-pp.fput_many
0.37 ± 8% -0.1 0.29 ± 14% perf-profile.self.cycles-pp.security_file_permission
0.15 ± 7% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.place_entity
0.18 ± 5% -0.1 0.10 ± 7% perf-profile.self.cycles-pp.__wake_up_common
0.17 ± 2% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.sched_clock_cpu
0.16 ± 4% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.__might_fault
0.20 ± 7% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.__list_del_entry_valid
0.16 ± 5% -0.1 0.09 perf-profile.self.cycles-pp.__list_add_valid
0.18 ± 5% -0.1 0.11 ± 3% perf-profile.self.cycles-pp.aa_file_perm
0.18 ± 3% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.put_prev_entity
0.16 ± 2% -0.1 0.10 perf-profile.self.cycles-pp.__wake_up_common_lock
0.11 ± 4% -0.1 0.05 perf-profile.self.cycles-pp.__bitmap_and
0.16 ± 2% -0.1 0.11 perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.11 -0.1 0.06 ± 7% perf-profile.self.cycles-pp.cpumask_next
0.09 ± 4% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.check_cfs_rq_runtime
0.13 ± 6% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.ktime_get
0.11 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.finish_wait
0.07 -0.0 0.03 ±100% perf-profile.self.cycles-pp.rw_verify_area
0.09 ± 4% -0.0 0.05 perf-profile.self.cycles-pp.ttwu_do_activate
0.39 -0.0 0.35 ± 3% perf-profile.self.cycles-pp.finish_task_switch
0.12 -0.0 0.08 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.51 -0.0 0.47 perf-profile.self.cycles-pp.__pthread_enable_asynccancel
0.09 -0.0 0.06 perf-profile.self.cycles-pp.sched_clock
0.20 ± 4% -0.0 0.17 perf-profile.self.cycles-pp.deactivate_task
0.15 ± 2% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.activate_task
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.__vfs_write
0.10 ± 14% +0.0 0.14 ± 9% perf-profile.self.cycles-pp.rb_insert_color
0.07 ± 7% +0.0 0.10 perf-profile.self.cycles-pp.rb_next
0.00 +0.1 0.05 perf-profile.self.cycles-pp.tracing_record_taskinfo_sched_switch
0.00 +0.1 0.05 perf-profile.self.cycles-pp.hrtick_update
0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.trace_clock_local
0.00 +0.1 0.07 perf-profile.self.cycles-pp.tracing_record_taskinfo_skip
0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.__trace_bprintk
0.00 +0.1 0.08 perf-profile.self.cycles-pp.ring_buffer_nest_end
0.17 +0.1 0.27 ± 6% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.00 +0.1 0.13 ± 3% perf-profile.self.cycles-pp.tracing_generic_entry_update
0.00 +0.1 0.14 ± 3% perf-profile.self.cycles-pp.ring_buffer_unlock_commit
0.00 +0.2 0.20 ± 5% perf-profile.self.cycles-pp.ring_buffer_nest_start
0.92 ± 4% +0.2 1.11 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.00 +0.2 0.21 perf-profile.self.cycles-pp.rb_commit
0.00 +0.3 0.27 perf-profile.self.cycles-pp.__rb_reserve_next
0.00 +0.9 0.88 ± 2% perf-profile.self.cycles-pp.vbin_printf
0.00 +1.0 1.04 ± 2% perf-profile.self.cycles-pp.ring_buffer_lock_reserve
0.00 +1.0 1.04 ± 4% perf-profile.self.cycles-pp.trace_vbprintk
0.00 +2.1 2.15 ± 2% perf-profile.self.cycles-pp.format_decode
0.00 +3.4 3.36 perf-profile.self.cycles-pp.probe_sched_switch
0.00 +3.4 3.40 perf-profile.self.cycles-pp.probe_sched_wakeup
0.00 +23.5 23.51 perf-profile.self.cycles-pp.trace_save_cmdline

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

Thanks,
Rong Chen

#
# Automatically generated file; DO NOT EDIT. # Linux/x86_64 5.8.0-rc1 Kernel Configuration #
CONFIG_CC_VERSION_TEXT="gcc-9 (Debian 9.3.0-13) 9.3.0" CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=90300 CONFIG_LD_VERSION=234000000 CONFIG_CLANG_VERSION=0
CONFIG_CC_CAN_LINK=y CONFIG_CC_CAN_LINK_STATIC=y CONFIG_CC_HAS_ASM_GOTO=y CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_IRQ_WORK=y CONFIG_BUILDTIME_TABLE_SORT=y CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# CONFIG_WATCH_QUEUE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_USELIB=y
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_MIGRATION=y
CONFIG_GENERIC_IRQ_INJECTION=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_IRQ_MSI_IOMMU=y
CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR=y
CONFIG_GENERIC_IRQ_RESERVATION_MODE=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set

# end of IRQ subsystem

CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_INIT=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
CONFIG_CONTEXT_TRACKING=y
# CONFIG_CONTEXT_TRACKING_FORCE is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

# end of Timers subsystem

# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_HAVE_SCHED_AVG_IRQ=y
# CONFIG_SCHED_THERMAL_PRESSURE is not set
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
# CONFIG_PSI is not set

# end of CPU/Task time and stats accounting

CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU_GENERIC=y
CONFIG_TASKS_RCU=y
CONFIG_TASKS_RUDE_RCU=y
CONFIG_TASKS_TRACE_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_RCU_NOCB_CPU=y

# end of RCU Subsystem

CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=20
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

#
# Scheduler features
#
# CONFIG_UCLAMP_TASK is not set

# end of Scheduler features

CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_CC_HAS_INT128=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_CGROUPS=y
CONFIG_PAGE_COUNTER=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_KMEM=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
# CONFIG_CGROUP_BPF is not set
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_TIME_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
# CONFIG_BOOT_CONFIG is not set
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BPF=y
# CONFIG_EXPERT is not set
CONFIG_UID16=y
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
# CONFIG_BPF_LSM is not set
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_USERFAULTFD=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_RSEQ=y
CONFIG_SCHED_PAIR_CPU=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set

# end of Kernel Performance Events And Counters

CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SYSTEM_DATA_VERIFICATION=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y

# end of General setup

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_FILTER_PGPROT=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_DYNAMIC_PHYSICAL_MASK=y
CONFIG_PGTABLE_LEVELS=5
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
CONFIG_RETPOLINE=y
CONFIG_X86_CPU_RESCTRL=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_NUMACHIP is not set
# CONFIG_X86_VSMP is not set
CONFIG_X86_UV=y
# CONFIG_X86_GOLDFISH is not set
# CONFIG_X86_INTEL_MID is not set
CONFIG_X86_INTEL_LPSS=y
CONFIG_X86_AMD_PLATFORM_DEVICE=y
CONFIG_IOSF_MBI=y
# CONFIG_IOSF_MBI_DEBUG is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_XXL=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_X86_HV_CALLBACK_VECTOR=y
CONFIG_XEN=y
CONFIG_XEN_PV=y
CONFIG_XEN_PV_SMP=y
# CONFIG_XEN_DOM0 is not set
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_PVH is not set
CONFIG_KVM_GUEST=y
CONFIG_ARCH_CPUIDLE_HALTPOLL=y
# CONFIG_PVH is not set
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_JAILHOUSE_GUEST is not set
# CONFIG_ACRN_GUEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_IA32_FEAT_CTL=y
CONFIG_X86_VMX_FEATURE_NAMES=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_HYGON=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_CPU_SUP_ZHAOXIN=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_MAXSMP=y
CONFIG_NR_CPUS_RANGE_BEGIN=8192
CONFIG_NR_CPUS_RANGE_END=8192
CONFIG_NR_CPUS_DEFAULT=8192
CONFIG_NR_CPUS=8192
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCELOG_LEGACY=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y

#
# Performance monitoring
#
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=y
CONFIG_PERF_EVENTS_INTEL_CSTATE=y
# CONFIG_PERF_EVENTS_AMD_POWER is not set

# end of Performance monitoring

CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y CONFIG_X86_VSYSCALL_EMULATION=y CONFIG_X86_IOPL_IOPERM=y
CONFIG_I8K=m
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y CONFIG_MICROCODE_OLD_INTERFACE=y CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_X86_5LEVEL=y
CONFIG_X86_DIRECT_GBPAGES=y
# CONFIG_X86_CPA_STATISTICS is not set CONFIG_AMD_MEM_ENCRYPT=y
# CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NUMA_EMU=y
CONFIG_NODES_SHIFT=10 CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SPARSEMEM_DEFAULT=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ARCH_MEMORY_PROBE=y CONFIG_ARCH_PROC_KCORE_TEXT=y CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000 CONFIG_X86_PMEM_LEGACY_DEVICE=y CONFIG_X86_PMEM_LEGACY=m CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1 CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1 CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
CONFIG_X86_UMIP=y CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y CONFIG_X86_INTEL_TSX_MODE_OFF=y
# CONFIG_X86_INTEL_TSX_MODE_ON is not set
# CONFIG_X86_INTEL_TSX_MODE_AUTO is not set CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_EFI_MIXED=y
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
# CONFIG_KEXEC_SIG is not set CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y CONFIG_PHYSICAL_START=0x1000000 CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y CONFIG_PHYSICAL_ALIGN=0x200000 CONFIG_DYNAMIC_MEMORY_LAYOUT=y CONFIG_RANDOMIZE_MEMORY=y CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa CONFIG_HOTPLUG_CPU=y CONFIG_BOOTPARAM_HOTPLUG_CPU0=y
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set CONFIG_LEGACY_VSYSCALL_EMULATE=y
# CONFIG_LEGACY_VSYSCALL_XONLY is not set
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set CONFIG_MODIFY_LDT_SYSCALL=y CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y

# end of Processor type and features

CONFIG_ARCH_HAS_ADD_PAGES=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_ENABLE_THP_MIGRATION=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_HIBERNATION_SNAPSHOT_DEV=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_PM_SLEEP_DEBUG=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_CLK=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
# CONFIG_ENERGY_MODEL is not set
CONFIG_ARCH_SUPPORTS_ACPI=y
CONFIG_ACPI=y
CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
# CONFIG_ACPI_DEBUGGER is not set
CONFIG_ACPI_SPCR_TABLE=y
CONFIG_ACPI_LPIT=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_TAD is not set
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_CPU_FREQ_PSS=y
CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ACPI_PROCESSOR_IDLE=y
CONFIG_ACPI_CPPC_LIB=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_HOTPLUG_IOAPIC=y
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
CONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_ACPI_BGRT=y
CONFIG_ACPI_NFIT=m
# CONFIG_NFIT_SECURITY_DEBUG is not set
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_HMAT is not set
CONFIG_HAVE_ACPI_APEI=y
CONFIG_HAVE_ACPI_APEI_NMI=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_ACPI_APEI_EINJ=m
CONFIG_ACPI_APEI_ERST_DEBUG=y
# CONFIG_DPTF_POWER is not set
CONFIG_ACPI_WATCHDOG=y
CONFIG_ACPI_EXTLOG=m
CONFIG_ACPI_ADXL=y
# CONFIG_PMIC_OPREGION is not set
# CONFIG_ACPI_CONFIGFS is not set
CONFIG_X86_PM_TIMER=y
CONFIG_SFI=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y CONFIG_CPU_FREQ_GOV_ATTR_SET=y CONFIG_CPU_FREQ_GOV_COMMON=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y

#
# CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_P4_CLOCKMOD=m

#

# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
# end of CPU Frequency scaling

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_CPU_IDLE_GOV_TEO is not set
# CONFIG_CPU_IDLE_GOV_HALTPOLL is not set
CONFIG_HALTPOLL_CPUIDLE=y

# end of CPU Idle

CONFIG_INTEL_IDLE=y

# end of Power management and ACPI options

#
# Bus options (PCI etc.)
#
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_MMCONF_FAM10H=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
# CONFIG_X86_SYSFB is not set

# end of Bus options (PCI etc.)

#
# Binary Emulations
#
CONFIG_IA32_EMULATION=y
# CONFIG_X86_X32 is not set
CONFIG_COMPAT_32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y

# end of Binary Emulations

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
CONFIG_FW_CFG_SYSFS=y
# CONFIG_FW_CFG_SYSFS_CMDLINE is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# EFI (Extensible Firmware Interface) Support
#
CONFIG_EFI_VARS=y
CONFIG_EFI_ESRT=y
CONFIG_EFI_VARS_PSTORE=y
CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y
CONFIG_EFI_RUNTIME_MAP=y
# CONFIG_EFI_FAKE_MEMMAP is not set
CONFIG_EFI_RUNTIME_WRAPPERS=y
CONFIG_EFI_GENERIC_STUB_INITRD_CMDLINE_LOADER=y
# CONFIG_EFI_BOOTLOADER_CONTROL is not set
# CONFIG_EFI_CAPSULE_LOADER is not set
# CONFIG_EFI_TEST is not set
CONFIG_APPLE_PROPERTIES=y
# CONFIG_RESET_ATTACK_MITIGATION is not set
# CONFIG_EFI_RCI2_TABLE is not set
# CONFIG_EFI_DISABLE_PCI_DMA is not set

# end of EFI (Extensible Firmware Interface) Support

CONFIG_UEFI_CPER=y
CONFIG_UEFI_CPER_X86=y
CONFIG_EFI_DEV_PATH_PARSER=y
CONFIG_EFI_EARLYCON=y

#
# Tegra firmware driver
#

# end of Tegra firmware driver
# end of Firmware Drivers

CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_HAVE_KVM_NO_POLL=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_AMD_SEV=y
CONFIG_KVM_MMU_AUDIT=y
CONFIG_AS_AVX512=y
CONFIG_AS_SHA1_NI=y
CONFIG_AS_SHA256_NI=y
CONFIG_AS_TPAUSE=y

#
# General architecture-dependent options
#
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_HOTPLUG_SMT=y
CONFIG_OPROFILE=m
CONFIG_OPROFILE_EVENT_MULTIPLEX=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
CONFIG_HAVE_SCHED_PAIR_CPU=y
CONFIG_HAVE_FUNCTION_ARG_ACCESS_API=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_MMU_GATHER_TABLE_FREE=y
CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_STACKPROTECTOR=y
CONFIG_STACKPROTECTOR_STRONG=y
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_STACK_VALIDATION=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
CONFIG_COMPAT_32BIT_TIME=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_HAVE_ARCH_PREL32_RELOCATIONS=y
CONFIG_ARCH_USE_MEMREMAP_PROT=y
# CONFIG_LOCK_EVENT_COUNTS is not set
CONFIG_ARCH_HAS_MEM_ENCRYPT=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y

# end of GCOV-based kernel profiling

CONFIG_HAVE_GCC_PLUGINS=y

# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULE_SIG_FORMAT=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set CONFIG_MODULE_SIG_ALL=y
# CONFIG_MODULE_SIG_SHA1 is not set
# CONFIG_MODULE_SIG_SHA224 is not set CONFIG_MODULE_SIG_SHA256=y
# CONFIG_MODULE_SIG_SHA384 is not set
# CONFIG_MODULE_SIG_SHA512 is not set CONFIG_MODULE_SIG_HASH="sha256"
# CONFIG_MODULE_COMPRESS is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set # CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_TRIM_UNUSED_KSYMS is not set CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_SCSI_REQUEST=y CONFIG_BLK_CGROUP_RWSTAT=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_INTEGRITY_T10=m CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_DEV_THROTTLING_LOW is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
# CONFIG_BLK_WBT is not set
# CONFIG_BLK_CGROUP_IOLATENCY is not set
# CONFIG_BLK_CGROUP_IOCOST is not set CONFIG_BLK_DEBUG_FS=y
CONFIG_BLK_DEBUG_FS_ZONED=y
# CONFIG_BLK_SED_OPAL is not set
# CONFIG_BLK_INLINE_ENCRYPTION is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set

# end of Partition Types

CONFIG_BLOCK_COMPAT=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_BLK_MQ_RDMA=y
CONFIG_BLK_PM=y

#
# IO Schedulers
#
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
# CONFIG_IOSCHED_BFQ is not set

# end of IO Schedulers

CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=y
CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE=y
CONFIG_ARCH_HAS_SYSCALL_WRAPPER=y
CONFIG_FREEZER=y

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y

# end of Executable file formats

#
# Memory Management options
#
CONFIG_SELECT_MEMORY_MODEL=y CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y CONFIG_NEED_MULTIPLE_NODES=y CONFIG_HAVE_MEMORY_PRESENT=y CONFIG_SPARSEMEM_EXTREME=y CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y CONFIG_SPARSEMEM_VMEMMAP=y CONFIG_HAVE_FAST_GUP=y CONFIG_NUMA_KEEP_MEMINFO=y CONFIG_MEMORY_ISOLATION=y CONFIG_HAVE_BOOTMEM_INFO_NODE=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG_SPARSE=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set CONFIG_MEMORY_HOTREMOVE=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_MEMORY_BALLOON=y CONFIG_BALLOON_COMPACTION=y CONFIG_COMPACTION=y
CONFIG_PAGE_REPORTING=y
CONFIG_MIGRATION=y
CONFIG_CONTIG_ALLOC=y CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y CONFIG_MEMORY_FAILURE=y CONFIG_HWPOISON_INJECT=m CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set CONFIG_ARCH_WANTS_THP_SWAP=y CONFIG_THP_SWAP=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
# CONFIG_CMA_DEBUGFS is not set CONFIG_CMA_AREAS=7
CONFIG_MEM_SOFT_DIRTY=y
CONFIG_ZSWAP=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set # CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo" CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set # CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
# CONFIG_ZSWAP_DEFAULT_ON is not set CONFIG_ZPOOL=y
CONFIG_ZBUD=y
# CONFIG_Z3FOLD is not set
CONFIG_ZSMALLOC=y
# CONFIG_ZSMALLOC_PGTABLE_MAPPING is not set # CONFIG_ZSMALLOC_STAT is not set CONFIG_GENERIC_EARLY_IOREMAP=y CONFIG_DEFERRED_STRUCT_PAGE_INIT=y CONFIG_IDLE_PAGE_TRACKING=y CONFIG_ARCH_HAS_PTE_DEVMAP=y CONFIG_ZONE_DEVICE=y CONFIG_DEV_PAGEMAP_OPS=y
# CONFIG_DEVICE_PRIVATE is not set CONFIG_FRAME_VECTOR=y CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y CONFIG_ARCH_HAS_PKEYS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_BENCHMARK is not set
# CONFIG_READ_ONLY_THP_FOR_FS is not set CONFIG_ARCH_HAS_PTE_SPECIAL=y CONFIG_MAPPING_DIRTY_HELPERS=y

# end of Memory Management options

CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_NET_REDIRECT=y
CONFIG_SKB_EXTENSIONS=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_SCM=y
CONFIG_UNIX_DIAG=m
# CONFIG_TLS is not set
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_INTERFACE is not set
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
# CONFIG_SMC is not set
# CONFIG_XDP_SOCKETS is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IP_TUNNEL=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_NET_UDP_TUNNEL=m
# CONFIG_NET_FOU is not set
# CONFIG_NET_FOU_IP_TUNNELS is not set
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
# CONFIG_INET_ESPINTCP is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
# CONFIG_INET_RAW_DIAG is not set
# CONFIG_INET_DIAG_DESTROY is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
# CONFIG_TCP_CONG_NV is not set
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_TCP_CONG_DCTCP=m
# CONFIG_TCP_CONG_CDG is not set
# CONFIG_TCP_CONG_BBR is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
# CONFIG_INET6_ESP_OFFLOAD is not set
# CONFIG_INET6_ESPINTCP is not set
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
# CONFIG_IPV6_ILA is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_VTI=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_GRE=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
# CONFIG_IPV6_SEG6_LWTUNNEL is not set
# CONFIG_IPV6_SEG6_HMAC is not set
# CONFIG_IPV6_RPL_LWTUNNEL is not set
CONFIG_NETLABEL=y
# CONFIG_MPTCP is not set
CONFIG_NETWORK_SECMARK=y
CONFIG_NET_PTP_CLASSIFY=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=m

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_FAMILY_BRIDGE=y
CONFIG_NETFILTER_FAMILY_ARP=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NETFILTER_NETLINK_OSF=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_COMMON=m
# CONFIG_NF_LOG_NETDEV is not set
CONFIG_NETFILTER_CONNCOUNT=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CONNTRACK_LABELS=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NF_CT_NETLINK_TIMEOUT=m
# CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_REDIRECT=y
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_NETFILTER_SYNPROXY=m
CONFIG_NF_TABLES=m
# CONFIG_NF_TABLES_INET is not set
# CONFIG_NF_TABLES_NETDEV is not set
# CONFIG_NFT_NUMGEN is not set
CONFIG_NFT_CT=m
CONFIG_NFT_COUNTER=m
# CONFIG_NFT_CONNLIMIT is not set
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_MASQ=m
CONFIG_NFT_REDIR=m
# CONFIG_NFT_TUNNEL is not set
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
# CONFIG_NFT_QUOTA is not set
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
# CONFIG_NFT_XFRM is not set
# CONFIG_NFT_SOCKET is not set
# CONFIG_NFT_OSF is not set
# CONFIG_NFT_TPROXY is not set
# CONFIG_NFT_SYNPROXY is not set
# CONFIG_NF_FLOW_TABLE is not set
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_NAT=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NETFILTER_XT_MATCH_CGROUP=m
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_L2TP=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m

# end of Core Netfilter Configuration

CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_IP_SET_HASH_IPMARK=m
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
CONFIG_IP_SET_HASH_IPMAC=m
CONFIG_IP_SET_HASH_MAC=m
CONFIG_IP_SET_HASH_NETPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_HASH_NETNET=m
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
# CONFIG_IP_VS_FO is not set
# CONFIG_IP_VS_OVF is not set
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
# CONFIG_IP_VS_MH is not set
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS MH scheduler
#
CONFIG_IP_VS_MH_TAB_INDEX=12

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_SOCKET_IPV4=m
CONFIG_NF_TPROXY_IPV4=m
# CONFIG_NF_TABLES_IPV4 is not set
# CONFIG_NF_TABLES_ARP is not set
CONFIG_NF_DUP_IPV4=m
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_SYNPROXY=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

# end of IP: Netfilter Configuration

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_SOCKET_IPV6=m
CONFIG_NF_TPROXY_IPV6=m
# CONFIG_NF_TABLES_IPV6 is not set
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
# CONFIG_IP6_NF_MATCH_SRH is not set
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_TARGET_SYNPROXY=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_IP6_NF_NAT=m
CONFIG_IP6_NF_TARGET_MASQUERADE=m
CONFIG_IP6_NF_TARGET_NPT=m

# end of IPv6: Netfilter Configuration

CONFIG_NF_DEFRAG_IPV6=m
# CONFIG_NF_TABLES_BRIDGE is not set
# CONFIG_NF_CONNTRACK_BRIDGE is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
# CONFIG_BPFILTER is not set
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y

# end of DCCP CCIDs Configuration

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set

# end of DCCP Kernel Hacking

CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5 is not set CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set CONFIG_SCTP_COOKIE_HMAC_MD5=y CONFIG_SCTP_COOKIE_HMAC_SHA1=y CONFIG_INET_SCTP_DIAG=m
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_MRP=m
CONFIG_BRIDGE=m CONFIG_BRIDGE_IGMP_SNOOPING=y CONFIG_BRIDGE_VLAN_FILTERING=y
# CONFIG_BRIDGE_MRP is not set CONFIG_HAVE_NET_DSA=y
# CONFIG_NET_DSA is not set CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y CONFIG_VLAN_8021Q_MVRP=y
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
CONFIG_6LOWPAN=m
# CONFIG_6LOWPAN_DEBUGFS is not set CONFIG_6LOWPAN_NHC=m CONFIG_6LOWPAN_NHC_DEST=m CONFIG_6LOWPAN_NHC_FRAGMENT=m CONFIG_6LOWPAN_NHC_HOP=m CONFIG_6LOWPAN_NHC_IPV6=m CONFIG_6LOWPAN_NHC_MOBILITY=m CONFIG_6LOWPAN_NHC_ROUTING=m CONFIG_6LOWPAN_NHC_UDP=m
# CONFIG_6LOWPAN_GHC_EXT_HDR_HOP is not set # CONFIG_6LOWPAN_GHC_UDP is not set
# CONFIG_6LOWPAN_GHC_ICMPV6 is not set
# CONFIG_6LOWPAN_GHC_EXT_HDR_DEST is not set # CONFIG_6LOWPAN_GHC_EXT_HDR_FRAG is not set # CONFIG_6LOWPAN_GHC_EXT_HDR_ROUTE is not set CONFIG_IEEE802154=m
# CONFIG_IEEE802154_NL802154_EXPERIMENTAL is not set CONFIG_IEEE802154_SOCKET=m CONFIG_IEEE802154_6LOWPAN=m CONFIG_MAC802154=m
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
# CONFIG_NET_SCH_CBS is not set
# CONFIG_NET_SCH_ETF is not set
# CONFIG_NET_SCH_TAPRIO is not set
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
# CONFIG_NET_SCH_SKBPRIO is not set
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
# CONFIG_NET_SCH_CAKE is not set
CONFIG_NET_SCH_FQ=m
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m
# CONFIG_NET_SCH_ETS is not set
# CONFIG_NET_SCH_DEFAULT is not set

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_FLOWER=m
CONFIG_NET_CLS_MATCHALL=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
# CONFIG_NET_EMATCH_CANID is not set
CONFIG_NET_EMATCH_IPSET=m
# CONFIG_NET_EMATCH_IPT is not set
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_SAMPLE=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
# CONFIG_NET_ACT_MPLS is not set
CONFIG_NET_ACT_VLAN=m
# CONFIG_NET_ACT_BPF is not set
CONFIG_NET_ACT_CONNMARK=m
# CONFIG_NET_ACT_CTINFO is not set
CONFIG_NET_ACT_SKBMOD=m
# CONFIG_NET_ACT_IFE is not set
CONFIG_NET_ACT_TUNNEL_KEY=m
# CONFIG_NET_ACT_GATE is not set
# CONFIG_NET_TC_SKB_EXT is not set
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=m
# CONFIG_BATMAN_ADV is not set
CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_GRE=m
CONFIG_OPENVSWITCH_VXLAN=m
CONFIG_OPENVSWITCH_GENEVE=m
CONFIG_VSOCKETS=m
CONFIG_VSOCKETS_DIAG=m
CONFIG_VSOCKETS_LOOPBACK=m
CONFIG_VMWARE_VMCI_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
CONFIG_HYPERV_VSOCKETS=m
CONFIG_NETLINK_DIAG=m
CONFIG_MPLS=y
CONFIG_NET_MPLS_GSO=y
# CONFIG_MPLS_ROUTING is not set
CONFIG_NET_NSH=m
# CONFIG_HSR is not set
CONFIG_NET_SWITCHDEV=y
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_QRTR is not set
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_CGROUP_NET_PRIO is not set
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_BPF_JIT=y
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
CONFIG_NET_DROP_MONITOR=y

# end of Network testing
# end of Networking options

# CONFIG_HAMRADIO is not set
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
CONFIG_CAN_GW=m
# CONFIG_CAN_J1939 is not set

#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_VXCAN is not set
CONFIG_CAN_SLCAN=m
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
# CONFIG_CAN_KVASER_PCIEFD is not set
CONFIG_CAN_C_CAN=m
CONFIG_CAN_C_CAN_PLATFORM=m
CONFIG_CAN_C_CAN_PCI=m
CONFIG_CAN_CC770=m
# CONFIG_CAN_CC770_ISA is not set
CONFIG_CAN_CC770_PLATFORM=m
# CONFIG_CAN_IFI_CANFD is not set
# CONFIG_CAN_M_CAN is not set
# CONFIG_CAN_PEAK_PCIEFD is not set
CONFIG_CAN_SJA1000=m
CONFIG_CAN_EMS_PCI=m
# CONFIG_CAN_F81601 is not set
CONFIG_CAN_KVASER_PCI=m
CONFIG_CAN_PEAK_PCI=m
CONFIG_CAN_PEAK_PCIEC=y
CONFIG_CAN_PLX_PCI=m
# CONFIG_CAN_SJA1000_ISA is not set
CONFIG_CAN_SJA1000_PLATFORM=m
CONFIG_CAN_SOFTING=m

#
# CAN SPI interfaces
#
# CONFIG_CAN_HI311X is not set
# CONFIG_CAN_MCP251X is not set

# end of CAN SPI interfaces

#
# CAN USB interfaces
#
CONFIG_CAN_8DEV_USB=m
CONFIG_CAN_EMS_USB=m
CONFIG_CAN_ESD_USB2=m
# CONFIG_CAN_GS_USB is not set
CONFIG_CAN_KVASER_USB=m
# CONFIG_CAN_MCBA_USB is not set
CONFIG_CAN_PEAK_USB=m
# CONFIG_CAN_UCAN is not set

# end of CAN USB interfaces

# CONFIG_CAN_DEBUG_DEVICES is not set

# end of CAN Device Drivers

CONFIG_BT=m
CONFIG_BT_BREDR=y
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HIDP=m
CONFIG_BT_HS=y
CONFIG_BT_LE=y
# CONFIG_BT_6LOWPAN is not set
# CONFIG_BT_LEDS is not set
# CONFIG_BT_MSFTEXT is not set
CONFIG_BT_DEBUGFS=y
# CONFIG_BT_SELFTEST is not set

#
# Bluetooth device drivers
#
CONFIG_BT_INTEL=m
CONFIG_BT_BCM=m
CONFIG_BT_RTL=m
CONFIG_BT_HCIBTUSB=m
# CONFIG_BT_HCIBTUSB_AUTOSUSPEND is not set
CONFIG_BT_HCIBTUSB_BCM=y
# CONFIG_BT_HCIBTUSB_MTK is not set
CONFIG_BT_HCIBTUSB_RTL=y
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
# CONFIG_BT_HCIUART_INTEL is not set
# CONFIG_BT_HCIUART_AG6XX is not set
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIVHCI=m
CONFIG_BT_MRVL=m
CONFIG_BT_MRVL_SDIO=m
CONFIG_BT_ATH3K=m
# CONFIG_BT_MTKSDIO is not set

# end of Bluetooth device drivers

# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y
CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
CONFIG_CFG80211_CRDA_SUPPORT=y
CONFIG_CFG80211_WEXT=y
CONFIG_LIB80211=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
# CONFIG_MAC80211_MESSAGE_TRACING is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_RFKILL_GPIO is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
# CONFIG_NET_9P_XEN is not set
# CONFIG_NET_9P_RDMA is not set
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
CONFIG_CEPH_LIB_USE_DNS_RESOLVER=y
# CONFIG_NFC is not set
CONFIG_PSAMPLE=m
# CONFIG_NET_IFE is not set
CONFIG_LWTUNNEL=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
CONFIG_NET_DEVLINK=y
CONFIG_PAGE_POOL=y
CONFIG_FAILOVER=m
CONFIG_ETHTOOL_NETLINK=y
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#
CONFIG_HAVE_EISA=y
# CONFIG_EISA is not set
CONFIG_HAVE_PCI=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIE_ECRC=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
# CONFIG_PCIE_DPC is not set
# CONFIG_PCIE_PTM is not set
# CONFIG_PCIE_BW is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
CONFIG_PCI_STUB=y
# CONFIG_PCI_PF_STUB is not set
# CONFIG_XEN_PCIDEV_FRONTEND is not set
CONFIG_PCI_ATS=y
CONFIG_PCI_LOCKLESS_CONFIG=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
# CONFIG_PCI_P2PDMA is not set
CONFIG_PCI_LABEL=y
CONFIG_PCI_HYPERV=m
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=y

#
# PCI controller drivers
#
CONFIG_VMD=y
CONFIG_PCI_HYPERV_INTERFACE=m

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT_HOST is not set
# CONFIG_PCI_MESON is not set

# end of DesignWare PCI Core Support

#
# Mobiveil PCIe Core Support
#

# end of Mobiveil PCIe Core Support

#
# Cadence PCIe controllers support
#

# end of Cadence PCIe controllers support
# end of PCI controller drivers

#
# PCI Endpoint
#
# CONFIG_PCI_ENDPOINT is not set

# end of PCI Endpoint

#
# PCI switch controller drivers
#
# CONFIG_PCI_SW_SWITCHTEC is not set

# end of PCI switch controller drivers

CONFIG_PCCARD=y
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
# CONFIG_RAPIDIO is not set

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

#
# Firmware loader
#
CONFIG_FW_LOADER=y CONFIG_FW_LOADER_PAGED_BUF=y CONFIG_EXTRA_FIRMWARE="" CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set # CONFIG_FW_LOADER_COMPRESS is not set CONFIG_FW_CACHE=y

# end of Firmware loader

CONFIG_WANT_DEV_COREDUMP=y
CONFIG_ALLOW_DEV_COREDUMP=y
CONFIG_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_PM_QOS_KUNIT_TEST is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_KUNIT_DRIVER_PE_TEST=y
CONFIG_SYS_HYPERVISOR=y
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=m
CONFIG_REGMAP_SPI=m
CONFIG_REGMAP_IRQ=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set

# end of Generic Driver Options

#
# Bus devices
#
# CONFIG_MHI_BUS is not set

# end of Bus devices

CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_GNSS is not set
CONFIG_MTD=m
# CONFIG_MTD_TESTS is not set

#
# Partition parsers
#
# CONFIG_MTD_AR7_PARTS is not set
# CONFIG_MTD_CMDLINE_PARTS is not set
# CONFIG_MTD_REDBOOT_PARTS is not set

# end of Partition parsers

#
# User Modules And Translation Layers
#
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
# CONFIG_MTD_BLOCK_RO is not set
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
# CONFIG_SSFDC is not set
# CONFIG_SM_FTL is not set
# CONFIG_MTD_OOPS is not set
# CONFIG_MTD_SWAP is not set
# CONFIG_MTD_PARTITIONED_MASTER is not set

#
# RAM/ROM/Flash chip drivers
#
# CONFIG_MTD_CFI is not set
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set

# end of RAM/ROM/Flash chip drivers

#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set

# end of Mapping drivers for chip access

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_DATAFLASH is not set
# CONFIG_MTD_MCHP23K256 is not set
# CONFIG_MTD_SST25L is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOCG3 is not set

# end of Self-contained MTD device drivers

# CONFIG_MTD_ONENAND is not set
# CONFIG_MTD_RAW_NAND is not set
# CONFIG_MTD_SPI_NAND is not set

#
# LPDDR & LPDDR2 PCM memory drivers
#
# CONFIG_MTD_LPDDR is not set

# end of LPDDR & LPDDR2 PCM memory drivers

# CONFIG_MTD_SPI_NOR is not set
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_BLOCK is not set
# CONFIG_MTD_HYPERBUS is not set
# CONFIG_OF is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION=y
CONFIG_BLK_DEV_FD=m
CONFIG_CDROM=m
# CONFIG_PARIDE is not set
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
# CONFIG_ZRAM is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SKD is not set
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_VIRTIO_BLK=y
CONFIG_BLK_DEV_RBD=m
# CONFIG_BLK_DEV_RSXX is not set

#
# NVME Support
#
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
# CONFIG_NVME_HWMON is not set
CONFIG_NVME_FABRICS=m
# CONFIG_NVME_RDMA is not set
CONFIG_NVME_FC=m
# CONFIG_NVME_TCP is not set
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
# CONFIG_NVME_TARGET_RDMA is not set
CONFIG_NVME_TARGET_FC=m
CONFIG_NVME_TARGET_FCLOOP=m
# CONFIG_NVME_TARGET_TCP is not set

# end of NVME Support

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=m
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_ICS932S401 is not set
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_SGI_XP=m
CONFIG_HP_ILO=m
CONFIG_SGI_GRU=m
# CONFIG_SGI_GRU_DEBUG is not set
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
CONFIG_VMWARE_BALLOON=m
# CONFIG_LATTICE_ECP3_CONFIG is not set
# CONFIG_SRAM is not set
# CONFIG_PCI_ENDPOINT_TEST is not set
# CONFIG_XILINX_SDFEC is not set
CONFIG_PVPANIC=y
# CONFIG_C2PORT is not set

#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
# CONFIG_EEPROM_AT25 is not set
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
# CONFIG_EEPROM_93XX46 is not set
# CONFIG_EEPROM_IDT_89HPESX is not set
# CONFIG_EEPROM_EE1004 is not set

# end of EEPROM support

CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y

#
# Texas Instruments shared transport line discipline #
# CONFIG_TI_ST is not set

# end of Texas Instruments shared transport line discipline

CONFIG_SENSORS_LIS3_I2C=m
CONFIG_ALTERA_STAPL=m
CONFIG_INTEL_MEI=m
CONFIG_INTEL_MEI_ME=m
# CONFIG_INTEL_MEI_TXE is not set
# CONFIG_INTEL_MEI_HDCP is not set
CONFIG_VMWARE_VMCI=m

#
# Intel MIC & related support
#
# CONFIG_INTEL_MIC_BUS is not set
# CONFIG_SCIF_BUS is not set
# CONFIG_VOP_BUS is not set

# end of Intel MIC & related support

# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
# CONFIG_MISC_ALCOR_PCI is not set
# CONFIG_MISC_RTSX_PCI is not set
# CONFIG_MISC_RTSX_USB is not set
# CONFIG_HABANA_AI is not set
# CONFIG_UACCE is not set

# end of Misc devices

CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_BLK_DEV_SR=m
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m

# end of SCSI Transports

CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_SCSI_BNX2X_FCOE=m
CONFIG_BE2ISCSI=m
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=m
# CONFIG_SCSI_AIC7XXX is not set
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC94XX is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_MVSAS_TASKLET=y
CONFIG_SCSI_MVUMI=m
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=m
# CONFIG_SCSI_SMARTPQI is not set
CONFIG_SCSI_UFSHCD=m
CONFIG_SCSI_UFSHCD_PCI=m
# CONFIG_SCSI_UFS_DWC_TC_PCI is not set
# CONFIG_SCSI_UFSHCD_PLATFORM is not set
# CONFIG_SCSI_UFS_BSG is not set
CONFIG_SCSI_HPTIOP=m
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_MYRB is not set
# CONFIG_SCSI_MYRS is not set
CONFIG_VMWARE_PVSCSI=m
# CONFIG_XEN_SCSI_FRONTEND is not set
CONFIG_HYPERV_STORAGE=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
# CONFIG_SCSI_SNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_FDOMAIN_PCI is not set
# CONFIG_SCSI_GDTH is not set
CONFIG_SCSI_ISCI=m
# CONFIG_SCSI_IPS is not set
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
CONFIG_SCSI_STEX=m
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=m
CONFIG_TCM_QLA2XXX=m
# CONFIG_TCM_QLA2XXX_DEBUG is not set
CONFIG_SCSI_QLA_ISCSI=m
# CONFIG_QEDI is not set
# CONFIG_QEDF is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
# CONFIG_SCSI_CHELSIO_FCOE is not set
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=y
CONFIG_SCSI_DH_HP_SW=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_ALUA=y

# end of SCSI device support

CONFIG_ATA=m
CONFIG_SATA_HOST=y
CONFIG_PATA_TIMINGS=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_FORCE=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_ZPODD is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
CONFIG_SATA_MOBILE_LPM_POLICY=0
CONFIG_SATA_AHCI_PLATFORM=m
# CONFIG_SATA_INIC162X is not set
CONFIG_SATA_ACARD_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
# CONFIG_SATA_DWC is not set
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=m
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
CONFIG_PATA_ACPI=m
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
# CONFIG_MD_CLUSTER is not set
# CONFIG_BCACHE is not set CONFIG_BLK_DEV_DM_BUILTIN=y CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set CONFIG_DM_BIO_PRISON=m CONFIG_DM_PERSISTENT_DATA=m
# CONFIG_DM_UNSTRIPED is not set CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m CONFIG_DM_THIN_PROVISIONING=m CONFIG_DM_CACHE=m
CONFIG_DM_CACHE_SMQ=m
# CONFIG_DM_WRITECACHE is not set
# CONFIG_DM_EBS is not set
CONFIG_DM_ERA=m
# CONFIG_DM_CLONE is not set CONFIG_DM_MIRROR=m CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_RAID=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m CONFIG_DM_MULTIPATH_QL=m CONFIG_DM_MULTIPATH_ST=m
# CONFIG_DM_MULTIPATH_HST is not set CONFIG_DM_DELAY=m
# CONFIG_DM_DUST is not set CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m
# CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG is not set # CONFIG_DM_VERITY_FEC is not set CONFIG_DM_SWITCH=m
CONFIG_DM_LOG_WRITES=m
# CONFIG_DM_INTEGRITY is not set
# CONFIG_DM_ZONED is not set CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_TCM_USER2=m CONFIG_LOOPBACK_TARGET=m
CONFIG_TCM_FC=m
CONFIG_ISCSI_TARGET=m CONFIG_ISCSI_TARGET_CXGB4=m
# CONFIG_SBP_TARGET is not set CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
# CONFIG_FUSION_FC is not set CONFIG_FUSION_SAS=m CONFIG_FUSION_MAX_SGE=128 CONFIG_FUSION_CTL=m CONFIG_FUSION_LOGGING=y

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
# CONFIG_FIREWIRE_NOSY is not set

# end of IEEE 1394 (FireWire) support

CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_WIREGUARD is not set
# CONFIG_EQUALIZER is not set
CONFIG_NET_FC=y
CONFIG_IFB=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
# CONFIG_IPVLAN is not set
CONFIG_VXLAN=m
CONFIG_GENEVE=m
# CONFIG_BAREUDP is not set
# CONFIG_GTP is not set
CONFIG_MACSEC=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_NTB_NETDEV=m
CONFIG_TUN=m
CONFIG_TAP=m
# CONFIG_TUN_VNET_CROSS_LE is not set
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m
CONFIG_NLMON=m
CONFIG_VSOCKMON=m
# CONFIG_ARCNET is not set
# CONFIG_ATM_DRIVERS is not set

#
# Distributed Switch Architecture drivers
#

# end of Distributed Switch Architecture drivers

CONFIG_ETHERNET=y
CONFIG_MDIO=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
CONFIG_NET_VENDOR_AGERE=y
# CONFIG_ET131X is not set
CONFIG_NET_VENDOR_ALACRITECH=y
# CONFIG_SLICOSS is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_AMAZON=y
CONFIG_ENA_ETHERNET=m
CONFIG_NET_VENDOR_AMD=y
CONFIG_AMD8111_ETH=m
CONFIG_PCNET32=m
CONFIG_AMD_XGBE=m
# CONFIG_AMD_XGBE_DCB is not set
CONFIG_AMD_XGBE_HAVE_ECC=y
CONFIG_NET_VENDOR_AQUANTIA=y
CONFIG_AQTION=m
CONFIG_NET_VENDOR_ARC=y
CONFIG_NET_VENDOR_ATHEROS=y
CONFIG_ATL2=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_ALX=m
CONFIG_NET_VENDOR_AURORA=y
# CONFIG_AURORA_NB8800 is not set
CONFIG_NET_VENDOR_BROADCOM=y
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
# CONFIG_BCMGENET is not set
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=y
CONFIG_TIGON3_HWMON=y
CONFIG_BNX2X=m
CONFIG_BNX2X_SRIOV=y
# CONFIG_SYSTEMPORT is not set
CONFIG_BNXT=m
CONFIG_BNXT_SRIOV=y
CONFIG_BNXT_FLOWER_OFFLOAD=y
CONFIG_BNXT_DCB=y
CONFIG_BNXT_HWMON=y
CONFIG_NET_VENDOR_BROCADE=y
CONFIG_BNA=m
CONFIG_NET_VENDOR_CADENCE=y
CONFIG_MACB=m
CONFIG_MACB_USE_HWSTAMP=y
# CONFIG_MACB_PCI is not set
CONFIG_NET_VENDOR_CAVIUM=y
# CONFIG_THUNDER_NIC_PF is not set
# CONFIG_THUNDER_NIC_VF is not set
# CONFIG_THUNDER_NIC_BGX is not set
# CONFIG_THUNDER_NIC_RGX is not set
CONFIG_CAVIUM_PTP=y
CONFIG_LIQUIDIO=m
CONFIG_LIQUIDIO_VF=m
CONFIG_NET_VENDOR_CHELSIO=y
# CONFIG_CHELSIO_T1 is not set
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
# CONFIG_CHELSIO_T4_DCB is not set
CONFIG_CHELSIO_T4VF=m
CONFIG_CHELSIO_LIB=m
CONFIG_NET_VENDOR_CISCO=y
CONFIG_ENIC=m
CONFIG_NET_VENDOR_CORTINA=y
# CONFIG_CX_ECAT is not set
CONFIG_DNET=m
CONFIG_NET_VENDOR_DEC=y
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=y
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_NET_VENDOR_DLINK is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=m
CONFIG_BE2NET_HWMON=y
CONFIG_BE2NET_BE2=y
CONFIG_BE2NET_BE3=y
CONFIG_BE2NET_LANCER=y
CONFIG_BE2NET_SKYHAWK=y
CONFIG_NET_VENDOR_EZCHIP=y
CONFIG_NET_VENDOR_GOOGLE=y
# CONFIG_GVE is not set
CONFIG_NET_VENDOR_HUAWEI=y
# CONFIG_HINIC is not set
# CONFIG_NET_VENDOR_I825XX is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_E1000E_HWTS=y
CONFIG_IGB=y
CONFIG_IGB_HWMON=y
CONFIG_IGBVF=m
# CONFIG_IXGB is not set
CONFIG_IXGBE=y
CONFIG_IXGBE_HWMON=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_I40E=y
CONFIG_I40E_DCB=y
CONFIG_IAVF=m
CONFIG_I40EVF=m
# CONFIG_ICE is not set
CONFIG_FM10K=m
# CONFIG_IGC is not set
CONFIG_JME=m
CONFIG_NET_VENDOR_MARVELL=y
CONFIG_MVMDIO=m
CONFIG_SKGE=y
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKGE_GENESIS=y
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
# CONFIG_MLX5_CORE is not set
# CONFIG_MLXSW_CORE is not set
# CONFIG_MLXFW is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
CONFIG_NET_VENDOR_MICROSEMI=y
# CONFIG_MSCC_OCELOT_SWITCH is not set
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
CONFIG_NET_VENDOR_NETERION=y
# CONFIG_S2IO is not set
# CONFIG_VXGE is not set
CONFIG_NET_VENDOR_NETRONOME=y
CONFIG_NFP=m
CONFIG_NFP_APP_FLOWER=y
CONFIG_NFP_APP_ABM_NIC=y
# CONFIG_NFP_DEBUG is not set
CONFIG_NET_VENDOR_NI=y
# CONFIG_NI_XGE_MANAGEMENT_ENET is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
CONFIG_NET_VENDOR_OKI=y
CONFIG_ETHOC=m
CONFIG_NET_VENDOR_PACKET_ENGINES=y
# CONFIG_HAMACHI is not set
CONFIG_YELLOWFIN=m
CONFIG_NET_VENDOR_PENSANDO=y
# CONFIG_IONIC is not set
CONFIG_NET_VENDOR_QLOGIC=y
CONFIG_QLA3XXX=m
CONFIG_QLCNIC=m
CONFIG_QLCNIC_SRIOV=y
CONFIG_QLCNIC_DCB=y
CONFIG_QLCNIC_HWMON=y
CONFIG_NETXEN_NIC=m
CONFIG_QED=m
CONFIG_QED_SRIOV=y
CONFIG_QEDE=m
CONFIG_NET_VENDOR_QUALCOMM=y
# CONFIG_QCOM_EMAC is not set
# CONFIG_RMNET is not set
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_ATP is not set
CONFIG_8139CP=y
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R8169=y
CONFIG_NET_VENDOR_RENESAS=y
CONFIG_NET_VENDOR_ROCKER=y
CONFIG_ROCKER=m
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
# CONFIG_NET_VENDOR_SEEQ is not set
CONFIG_NET_VENDOR_SOLARFLARE=y
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_SFC_MCDI_MON=y
CONFIG_SFC_SRIOV=y
CONFIG_SFC_MCDI_LOGGING=y
CONFIG_SFC_FALCON=m
CONFIG_SFC_FALCON_MTD=y
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
CONFIG_NET_VENDOR_SMSC=y
CONFIG_EPIC100=m
# CONFIG_SMSC911X is not set
CONFIG_SMSC9420=m
CONFIG_NET_VENDOR_SOCIONEXT=y
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
CONFIG_NET_VENDOR_SYNOPSYS=y
# CONFIG_DWC_XLGMAC is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
CONFIG_NET_VENDOR_TI=y
# CONFIG_TI_CPSW_PHY_SEL is not set
CONFIG_TLAN=m
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_NET_VENDOR_XILINX=y
# CONFIG_XILINX_AXI_EMAC is not set
# CONFIG_XILINX_LL_TEMAC is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
# CONFIG_MDIO_BCM_UNIMAC is not set
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
# CONFIG_MDIO_MSCC_MIIM is not set
# CONFIG_MDIO_MVUSB is not set
# CONFIG_MDIO_THUNDER is not set
# CONFIG_MDIO_XPCS is not set
CONFIG_PHYLINK=m
CONFIG_PHYLIB=y
CONFIG_SWPHY=y
# CONFIG_LED_TRIGGER_PHY is not set

#
# MII PHY device drivers
#
# CONFIG_SFP is not set
# CONFIG_ADIN_PHY is not set
CONFIG_AMD_PHY=m
# CONFIG_AQUANTIA_PHY is not set
# CONFIG_AX88796B_PHY is not set
# CONFIG_BCM7XXX_PHY is not set
CONFIG_BCM87XX_PHY=m
CONFIG_BCM_NET_PHYLIB=m
CONFIG_BROADCOM_PHY=m
# CONFIG_BCM54140_PHY is not set
# CONFIG_BCM84881_PHY is not set
CONFIG_CICADA_PHY=m
# CONFIG_CORTINA_PHY is not set
CONFIG_DAVICOM_PHY=m
# CONFIG_DP83822_PHY is not set
# CONFIG_DP83TC811_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
# CONFIG_DP83869_PHY is not set
CONFIG_FIXED_PHY=y
CONFIG_ICPLUS_PHY=m
# CONFIG_INTEL_XWAY_PHY is not set
CONFIG_LSI_ET1011C_PHY=m
CONFIG_LXT_PHY=m
CONFIG_MARVELL_PHY=m
# CONFIG_MARVELL_10G_PHY is not set
CONFIG_MICREL_PHY=m
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROCHIP_T1_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
CONFIG_NATIONAL_PHY=m
# CONFIG_NXP_TJA11XX_PHY is not set
CONFIG_QSEMI_PHY=m
CONFIG_REALTEK_PHY=y
# CONFIG_RENESAS_PHY is not set
# CONFIG_ROCKCHIP_PHY is not set
CONFIG_SMSC_PHY=m
CONFIG_STE10XP=m
# CONFIG_TERANETICS_PHY is not set
CONFIG_VITESSE_PHY=m
# CONFIG_XILINX_GMII2RGMII is not set
# CONFIG_MICREL_KS8995MA is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOATM=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOL2TP=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_SLIP=m
CONFIG_SLHC=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_USB_NET_DRIVERS=y
CONFIG_USB_CATC=y
CONFIG_USB_KAWETH=y
CONFIG_USB_PEGASUS=y
CONFIG_USB_RTL8150=y
CONFIG_USB_RTL8152=m
# CONFIG_USB_LAN78XX is not set
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
CONFIG_USB_NET_AX88179_178A=m
CONFIG_USB_NET_CDCETHER=y
CONFIG_USB_NET_CDC_EEM=y
CONFIG_USB_NET_CDC_NCM=m
CONFIG_USB_NET_HUAWEI_CDC_NCM=m
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_DM9601=y
# CONFIG_USB_NET_SR9700 is not set
# CONFIG_USB_NET_SR9800 is not set
CONFIG_USB_NET_SMSC75XX=y
CONFIG_USB_NET_SMSC95XX=y
CONFIG_USB_NET_GL620A=y
CONFIG_USB_NET_NET1080=y
CONFIG_USB_NET_PLUSB=y
CONFIG_USB_NET_MCS7830=y
CONFIG_USB_NET_RNDIS_HOST=y
CONFIG_USB_NET_CDC_SUBSET_ENABLE=y
CONFIG_USB_NET_CDC_SUBSET=y
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=y
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_NET_KALMIA=m
CONFIG_USB_NET_QMI_WWAN=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=y
CONFIG_USB_IPHETH=y
CONFIG_USB_SIERRA_NET=y
CONFIG_USB_VL600=m
# CONFIG_USB_NET_CH9200 is not set
# CONFIG_USB_NET_AQC111 is not set
CONFIG_WLAN=y
CONFIG_WLAN_VENDOR_ADMTEK=y
# CONFIG_ADM8211 is not set
CONFIG_ATH_COMMON=m
CONFIG_WLAN_VENDOR_ATH=y
# CONFIG_ATH_DEBUG is not set
# CONFIG_ATH5K is not set
# CONFIG_ATH5K_PCI is not set
CONFIG_ATH9K_HW=m
CONFIG_ATH9K_COMMON=m
CONFIG_ATH9K_BTCOEX_SUPPORT=y
# CONFIG_ATH9K is not set
CONFIG_ATH9K_HTC=m
# CONFIG_ATH9K_HTC_DEBUGFS is not set
# CONFIG_CARL9170 is not set
# CONFIG_ATH6KL is not set
# CONFIG_AR5523 is not set
# CONFIG_WIL6210 is not set
# CONFIG_ATH10K is not set
# CONFIG_WCN36XX is not set
CONFIG_WLAN_VENDOR_ATMEL=y
# CONFIG_ATMEL is not set
# CONFIG_AT76C50X_USB is not set
CONFIG_WLAN_VENDOR_BROADCOM=y
# CONFIG_B43 is not set
# CONFIG_B43LEGACY is not set
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
CONFIG_WLAN_VENDOR_CISCO=y
# CONFIG_AIRO is not set
CONFIG_WLAN_VENDOR_INTEL=y
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
CONFIG_IWLEGACY=m
CONFIG_IWL4965=m
CONFIG_IWL3945=m

#

# iwl3945 / iwl4965 Debugging Options
#
CONFIG_IWLEGACY_DEBUG=y
CONFIG_IWLEGACY_DEBUGFS=y
# end of iwl3945 / iwl4965 Debugging Options

CONFIG_IWLWIFI=m
CONFIG_IWLWIFI_LEDS=y
CONFIG_IWLDVM=m
CONFIG_IWLMVM=m
CONFIG_IWLWIFI_OPMODE_MODULAR=y
# CONFIG_IWLWIFI_BCAST_FILTERING is not set

#
# Debugging Options
#
# CONFIG_IWLWIFI_DEBUG is not set
CONFIG_IWLWIFI_DEBUGFS=y
# CONFIG_IWLWIFI_DEVICE_TRACING is not set

# end of Debugging Options

CONFIG_WLAN_VENDOR_INTERSIL=y
# CONFIG_HOSTAP is not set
# CONFIG_HERMES is not set
# CONFIG_P54_COMMON is not set
# CONFIG_PRISM54 is not set
CONFIG_WLAN_VENDOR_MARVELL=y
# CONFIG_LIBERTAS is not set
# CONFIG_LIBERTAS_THINFIRM is not set
# CONFIG_MWIFIEX is not set
# CONFIG_MWL8K is not set
CONFIG_WLAN_VENDOR_MEDIATEK=y
# CONFIG_MT7601U is not set
# CONFIG_MT76x0U is not set
# CONFIG_MT76x0E is not set
# CONFIG_MT76x2E is not set
# CONFIG_MT76x2U is not set
# CONFIG_MT7603E is not set
# CONFIG_MT7615E is not set
# CONFIG_MT7663U is not set
# CONFIG_MT7915E is not set
CONFIG_WLAN_VENDOR_RALINK=y
# CONFIG_RT2X00 is not set
CONFIG_WLAN_VENDOR_REALTEK=y
# CONFIG_RTL8180 is not set
# CONFIG_RTL8187 is not set
# CONFIG_RTL_CARDS is not set
# CONFIG_RTL8XXXU is not set
# CONFIG_RTW88 is not set
CONFIG_WLAN_VENDOR_RSI=y
# CONFIG_RSI_91X is not set
CONFIG_WLAN_VENDOR_ST=y
# CONFIG_CW1200 is not set
CONFIG_WLAN_VENDOR_TI=y
# CONFIG_WL1251 is not set
# CONFIG_WL12XX is not set
# CONFIG_WL18XX is not set
# CONFIG_WLCORE is not set
CONFIG_WLAN_VENDOR_ZYDAS=y
# CONFIG_USB_ZD1201 is not set
# CONFIG_ZD1211RW is not set
CONFIG_WLAN_VENDOR_QUANTENNA=y
# CONFIG_QTNFMAC_PCIE is not set
CONFIG_MAC80211_HWSIM=m
# CONFIG_USB_NET_RNDIS_WLAN is not set
# CONFIG_VIRT_WIFI is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers #
CONFIG_WAN=y
# CONFIG_LANMEDIA is not set
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
# CONFIG_HDLC_RAW_ETH is not set CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m

#
# X.25/LAPB support is disabled
#
# CONFIG_PCI200SYN is not set
# CONFIG_WANXL is not set
# CONFIG_PC300TOO is not set
# CONFIG_FARSYNC is not set
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
# CONFIG_SBNI is not set
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKELB=m
# CONFIG_IEEE802154_AT86RF230 is not set
# CONFIG_IEEE802154_MRF24J40 is not set
# CONFIG_IEEE802154_CC2520 is not set
# CONFIG_IEEE802154_ATUSB is not set
# CONFIG_IEEE802154_ADF7242 is not set
# CONFIG_IEEE802154_CA8210 is not set
# CONFIG_IEEE802154_MCR20A is not set
# CONFIG_IEEE802154_HWSIM is not set
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_VMXNET3=m
CONFIG_FUJITSU_ES=m
CONFIG_HYPERV_NET=m
CONFIG_NETDEVSIM=m
CONFIG_NET_FAILOVER=m
CONFIG_ISDN=y
CONFIG_ISDN_CAPI=y
CONFIG_CAPI_TRACE=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m

#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
CONFIG_MISDN_NETJET=m
CONFIG_MISDN_HDLC=m
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_NVM=y
# CONFIG_NVM_PBLK is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_LEDS=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADC is not set
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
# CONFIG_KEYBOARD_APPLESPI is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1050 is not set
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_DLINK_DIR685 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_TM2_TOUCHKEY is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_BYD=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_SYNAPTICS_SMBUS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_ELANTECH_SMBUS=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_PS2_FOCALTECH=y
CONFIG_MOUSE_PS2_VMMOUSE=y
CONFIG_MOUSE_PS2_SMBUS=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_CYAPA=m
# CONFIG_MOUSE_ELAN_I2C is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_MOUSE_SYNAPTICS_USB=m
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
# CONFIG_TABLET_USB_HANWANG is not set
CONFIG_TABLET_USB_KBTAB=m
# CONFIG_TABLET_USB_PEGASUS is not set
# CONFIG_TABLET_SERIAL_WACOM4 is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_PROPERTIES=y
# CONFIG_TOUCHSCREEN_ADS7846 is not set
# CONFIG_TOUCHSCREEN_AD7877 is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_ADC is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_AUO_PIXCIR is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_BU21029 is not set
# CONFIG_TOUCHSCREEN_CHIPONE_ICN8505 is not set
# CONFIG_TOUCHSCREEN_CY8CTMA140 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
# CONFIG_TOUCHSCREEN_CYTTSP_CORE is not set
# CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_EGALAX_SERIAL is not set
# CONFIG_TOUCHSCREEN_EXC3000 is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GOODIX is not set
# CONFIG_TOUCHSCREEN_HIDEEP is not set
# CONFIG_TOUCHSCREEN_ILI210X is not set
# CONFIG_TOUCHSCREEN_S6SY761 is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_EKTF2127 is not set
# CONFIG_TOUCHSCREEN_ELAN is not set
CONFIG_TOUCHSCREEN_ELO=m
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_WACOM_I2C=m
# CONFIG_TOUCHSCREEN_MAX11801 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
# CONFIG_TOUCHSCREEN_MMS114 is not set
# CONFIG_TOUCHSCREEN_MELFAS_MIP4 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_PIXCIR is not set
# CONFIG_TOUCHSCREEN_WDT87XX_I2C is not set
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC_SERIO is not set
# CONFIG_TOUCHSCREEN_TSC2004 is not set
# CONFIG_TOUCHSCREEN_TSC2005 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_RM_TS is not set
# CONFIG_TOUCHSCREEN_SILEAD is not set
# CONFIG_TOUCHSCREEN_SIS_I2C is not set
# CONFIG_TOUCHSCREEN_ST1232 is not set
# CONFIG_TOUCHSCREEN_STMFTS is not set
# CONFIG_TOUCHSCREEN_SUR40 is not set
# CONFIG_TOUCHSCREEN_SURFACE3_SPI is not set
# CONFIG_TOUCHSCREEN_SX8654 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
# CONFIG_TOUCHSCREEN_ZET6223 is not set
# CONFIG_TOUCHSCREEN_ZFORCE is not set
# CONFIG_TOUCHSCREEN_ROHM_BU21023 is not set
# CONFIG_TOUCHSCREEN_IQS5XX is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_MMA8450 is not set
CONFIG_INPUT_APANEL=m
# CONFIG_INPUT_GPIO_BEEPER is not set
# CONFIG_INPUT_GPIO_DECODER is not set
# CONFIG_INPUT_GPIO_VIBRA is not set
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
# CONFIG_INPUT_KXTJ9 is not set
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_PWM_BEEPER is not set
# CONFIG_INPUT_PWM_VIBRA is not set
CONFIG_INPUT_GPIO_ROTARY_ENCODER=m
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_IQS269A is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_DRV260X_HAPTICS is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
CONFIG_RMI4_CORE=m
# CONFIG_RMI4_I2C is not set
# CONFIG_RMI4_SPI is not set
CONFIG_RMI4_SMB=m
CONFIG_RMI4_F03=y
CONFIG_RMI4_F03_SERIO=m
CONFIG_RMI4_2D_SENSOR=y
CONFIG_RMI4_F11=y
CONFIG_RMI4_F12=y
CONFIG_RMI4_F30=y
# CONFIG_RMI4_F34 is not set
# CONFIG_RMI4_F54 is not set
# CONFIG_RMI4_F55 is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_SERIO_ARC_PS2=m
CONFIG_HYPERV_KEYBOARD=m
# CONFIG_SERIO_GPIO_PS2 is not set
# CONFIG_USERIO is not set
# CONFIG_GAMEPORT is not set

# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_LDISC_AUTOLOAD=y

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set CONFIG_SERIAL_8250_PNP=y
# CONFIG_SERIAL_8250_16550A_VARIANTS is not set # CONFIG_SERIAL_8250_FINTEK is not set CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_DMA=y CONFIG_SERIAL_8250_PCI=y CONFIG_SERIAL_8250_EXAR=y CONFIG_SERIAL_8250_NR_UARTS=32 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set CONFIG_SERIAL_8250_RSA=y CONFIG_SERIAL_8250_DWLIB=y CONFIG_SERIAL_8250_DW=y
# CONFIG_SERIAL_8250_RT288X is not set CONFIG_SERIAL_8250_LPSS=y CONFIG_SERIAL_8250_MID=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_LANTIQ is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_SC16IS7XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
CONFIG_SERIAL_ARC=m
CONFIG_SERIAL_ARC_NR_PORTS=1
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_FSL_LINFLEXUART is not set
# CONFIG_SERIAL_SPRD is not set

# end of Serial drivers

CONFIG_SERIAL_MCTRL_GPIO=y
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
# CONFIG_ISI is not set
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
CONFIG_NOZOMI=m
# CONFIG_NULL_TTY is not set
# CONFIG_TRACE_SINK is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_SERIAL_DEV_BUS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DMI_DECODE=y
CONFIG_IPMI_PLAT_DATA=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_SSIF=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=y
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
CONFIG_DEVMEM=y
# CONFIG_DEVKMEM is not set
CONFIG_NVRAM=y
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_DEVPORT=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HPET_MMAP_DEFAULT is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_UV_MMTIMER=m
CONFIG_TCG_TPM=y
CONFIG_HW_RANDOM_TPM=y
CONFIG_TCG_TIS_CORE=y
CONFIG_TCG_TIS=y
# CONFIG_TCG_TIS_SPI is not set
CONFIG_TCG_TIS_I2C_ATMEL=m
CONFIG_TCG_TIS_I2C_INFINEON=m
CONFIG_TCG_TIS_I2C_NUVOTON=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TCG_XEN is not set
CONFIG_TCG_CRB=y
# CONFIG_TCG_VTPM_PROXY is not set
CONFIG_TCG_TIS_ST33ZP24=m
CONFIG_TCG_TIS_ST33ZP24_I2C=m
# CONFIG_TCG_TIS_ST33ZP24_SPI is not set
CONFIG_TELCLOCK=m
# CONFIG_XILLYBUS is not set

# end of Character devices

# CONFIG_RANDOM_TRUST_CPU is not set
# CONFIG_RANDOM_TRUST_BOOTLOADER is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_ACPI_I2C_OPREGION=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_MUX=m

#
# Multiplexer I2C Chip support
#
# CONFIG_I2C_MUX_GPIO is not set
# CONFIG_I2C_MUX_LTC4306 is not set
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
# CONFIG_I2C_MUX_REG is not set
# CONFIG_I2C_MUX_MLXCPLD is not set

# end of Multiplexer I2C Chip support

CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
# CONFIG_I2C_AMD_MP2 is not set
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
# CONFIG_I2C_NVIDIA_GPU is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
CONFIG_I2C_SCMI=m

#
# I2C system bus drivers (mostly embedded / system-on-chip) #
# CONFIG_I2C_CBUS_GPIO is not set CONFIG_I2C_DESIGNWARE_CORE=m
# CONFIG_I2C_DESIGNWARE_SLAVE is not set CONFIG_I2C_DESIGNWARE_PLATFORM=m
# CONFIG_I2C_DESIGNWARE_BAYTRAIL is not set
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_EMEV2 is not set
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_OCORES is not set CONFIG_I2C_PCA_PLATFORM=m
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_DIOLAN_U2C=m
CONFIG_I2C_PARPORT=m
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_VIPERBOARD=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_MLXCPLD is not set

# end of I2C Hardware Bus support

CONFIG_I2C_STUB=m
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set

# end of I2C support

# CONFIG_I3C is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y
# CONFIG_SPI_MEM is not set

#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
# CONFIG_SPI_AXI_SPI_ENGINE is not set
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_CADENCE is not set
# CONFIG_SPI_DESIGNWARE is not set
# CONFIG_SPI_NXP_FLEXSPI is not set
# CONFIG_SPI_GPIO is not set
# CONFIG_SPI_LM70_LLP is not set
# CONFIG_SPI_OC_TINY is not set
CONFIG_SPI_PXA2XX=m
CONFIG_SPI_PXA2XX_PCI=m
# CONFIG_SPI_ROCKCHIP is not set
# CONFIG_SPI_SC18IS602 is not set
# CONFIG_SPI_SIFIVE is not set
# CONFIG_SPI_MXIC is not set
# CONFIG_SPI_XCOMM is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_ZYNQMP_GQSPI is not set
# CONFIG_SPI_AMD is not set

#
# SPI Multiplexer support
#
# CONFIG_SPI_MUX is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_LOOPBACK_TEST is not set
# CONFIG_SPI_TLE62X0 is not set
# CONFIG_SPI_SLAVE is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
CONFIG_PPS_CLIENT_PARPORT=m
CONFIG_PPS_CLIENT_GPIO=m

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y
CONFIG_DP83640_PHY=m
# CONFIG_PTP_1588_CLOCK_INES is not set
CONFIG_PTP_1588_CLOCK_KVM=m
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# CONFIG_PTP_1588_CLOCK_VMW is not set

# end of PTP clock support

CONFIG_PINCTRL=y
CONFIG_PINMUX=y
CONFIG_PINCONF=y
CONFIG_GENERIC_PINCONF=y
# CONFIG_DEBUG_PINCTRL is not set
CONFIG_PINCTRL_AMD=m
# CONFIG_PINCTRL_MCP23S08 is not set
# CONFIG_PINCTRL_SX150X is not set
CONFIG_PINCTRL_BAYTRAIL=y
# CONFIG_PINCTRL_CHERRYVIEW is not set
# CONFIG_PINCTRL_LYNXPOINT is not set
CONFIG_PINCTRL_INTEL=m
# CONFIG_PINCTRL_BROXTON is not set
CONFIG_PINCTRL_CANNONLAKE=m
# CONFIG_PINCTRL_CEDARFORK is not set
CONFIG_PINCTRL_DENVERTON=m
CONFIG_PINCTRL_GEMINILAKE=m
# CONFIG_PINCTRL_ICELAKE is not set
# CONFIG_PINCTRL_JASPERLAKE is not set
CONFIG_PINCTRL_LEWISBURG=m
CONFIG_PINCTRL_SUNRISEPOINT=m
# CONFIG_PINCTRL_TIGERLAKE is not set
CONFIG_GPIOLIB=y
CONFIG_GPIOLIB_FASTPATH_LIMIT=512
CONFIG_GPIO_ACPI=y
CONFIG_GPIOLIB_IRQCHIP=y
# CONFIG_DEBUG_GPIO is not set
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_GENERIC=m

#
# Memory mapped GPIO drivers
#
CONFIG_GPIO_AMDPT=m
# CONFIG_GPIO_DWAPB is not set
# CONFIG_GPIO_EXAR is not set
# CONFIG_GPIO_GENERIC_PLATFORM is not set
CONFIG_GPIO_ICH=m
# CONFIG_GPIO_MB86S7X is not set
# CONFIG_GPIO_VX855 is not set
# CONFIG_GPIO_XILINX is not set
# CONFIG_GPIO_AMD_FCH is not set

# end of Memory mapped GPIO drivers

#
# Port-mapped I/O GPIO drivers
#
# CONFIG_GPIO_F7188X is not set
# CONFIG_GPIO_IT87 is not set
# CONFIG_GPIO_SCH is not set
# CONFIG_GPIO_SCH311X is not set
# CONFIG_GPIO_WINBOND is not set
# CONFIG_GPIO_WS16C48 is not set

# end of Port-mapped I/O GPIO drivers

#
# I2C GPIO expanders
#
# CONFIG_GPIO_ADP5588 is not set
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_TPIC2810 is not set

# end of I2C GPIO expanders

#
# MFD GPIO expanders
#

# end of MFD GPIO expanders

#
# PCI GPIO expanders
#
# CONFIG_GPIO_AMD8111 is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_PCI_IDIO_16 is not set
# CONFIG_GPIO_PCIE_IDIO_24 is not set
# CONFIG_GPIO_RDC321X is not set

# end of PCI GPIO expanders

#
# SPI GPIO expanders
#
# CONFIG_GPIO_MAX3191X is not set
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_PISOSR is not set
# CONFIG_GPIO_XRA1403 is not set

# end of SPI GPIO expanders

#
# USB GPIO expanders
#
CONFIG_GPIO_VIPERBOARD=m

# end of USB GPIO expanders

# CONFIG_GPIO_AGGREGATOR is not set
# CONFIG_GPIO_MOCKUP is not set
# CONFIG_W1 is not set
# CONFIG_POWER_AVS is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_RESET_RESTART is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_POWER_SUPPLY_HWMON=y
# CONFIG_PDA_POWER is not set
# CONFIG_GENERIC_ADC_BATTERY is not set
# CONFIG_TEST_POWER is not set
# CONFIG_CHARGER_ADP5061 is not set
# CONFIG_BATTERY_CW2015 is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_CHARGER_SBS is not set
# CONFIG_MANAGER_SBS is not set
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_GPIO is not set
# CONFIG_CHARGER_LT3651 is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_BQ24257 is not set
# CONFIG_CHARGER_BQ24735 is not set
# CONFIG_CHARGER_BQ25890 is not set
CONFIG_CHARGER_SMB347=m
# CONFIG_BATTERY_GAUGE_LTC2941 is not set
# CONFIG_CHARGER_RT9455 is not set
# CONFIG_CHARGER_BD99954 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
# CONFIG_SENSORS_AD7314 is not set
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
# CONFIG_SENSORS_ADM1177 is not set
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7X10=m
# CONFIG_SENSORS_ADT7310 is not set
CONFIG_SENSORS_ADT7410=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
# CONFIG_SENSORS_AS370 is not set
CONFIG_SENSORS_ASC7621=m
# CONFIG_SENSORS_AXI_FAN_CONTROL is not set
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_FAM15H_POWER=m
# CONFIG_SENSORS_AMD_ENERGY is not set
CONFIG_SENSORS_APPLESMC=m
CONFIG_SENSORS_ASB100=m
# CONFIG_SENSORS_ASPEED is not set
CONFIG_SENSORS_ATXP1=m
# CONFIG_SENSORS_DRIVETEMP is not set
CONFIG_SENSORS_DS620=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_DELL_SMM=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
# CONFIG_SENSORS_FTSTEUTATES is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_G760A=m
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_HIH6130 is not set
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
# CONFIG_SENSORS_IIO_HWMON is not set
# CONFIG_SENSORS_I5500 is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_JC42=m
# CONFIG_SENSORS_POWR1220 is not set
CONFIG_SENSORS_LINEAGE=m
# CONFIG_SENSORS_LTC2945 is not set
# CONFIG_SENSORS_LTC2947_I2C is not set
# CONFIG_SENSORS_LTC2947_SPI is not set
# CONFIG_SENSORS_LTC2990 is not set
CONFIG_SENSORS_LTC4151=m
CONFIG_SENSORS_LTC4215=m
# CONFIG_SENSORS_LTC4222 is not set
CONFIG_SENSORS_LTC4245=m
# CONFIG_SENSORS_LTC4260 is not set
CONFIG_SENSORS_LTC4261=m
# CONFIG_SENSORS_MAX1111 is not set
CONFIG_SENSORS_MAX16065=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX1668=m
CONFIG_SENSORS_MAX197=m
# CONFIG_SENSORS_MAX31722 is not set
# CONFIG_SENSORS_MAX31730 is not set
# CONFIG_SENSORS_MAX6621 is not set
CONFIG_SENSORS_MAX6639=m
CONFIG_SENSORS_MAX6642=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_MAX6697=m
# CONFIG_SENSORS_MAX31790 is not set
CONFIG_SENSORS_MCP3021=m
# CONFIG_SENSORS_TC654 is not set
# CONFIG_SENSORS_ADCXX is not set
CONFIG_SENSORS_LM63=m
# CONFIG_SENSORS_LM70 is not set
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LM95234=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_LM95245=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_NTC_THERMISTOR=m
# CONFIG_SENSORS_NCT6683 is not set
CONFIG_SENSORS_NCT6775=m
# CONFIG_SENSORS_NCT7802 is not set
# CONFIG_SENSORS_NCT7904 is not set
# CONFIG_SENSORS_NPCM7XX is not set
CONFIG_SENSORS_PCF8591=m
CONFIG_PMBUS=m
CONFIG_SENSORS_PMBUS=m
CONFIG_SENSORS_ADM1275=m
# CONFIG_SENSORS_BEL_PFE is not set
# CONFIG_SENSORS_IBM_CFFPS is not set
# CONFIG_SENSORS_INSPUR_IPSPS is not set
# CONFIG_SENSORS_IR35221 is not set
# CONFIG_SENSORS_IR38064 is not set
# CONFIG_SENSORS_IRPS5401 is not set
# CONFIG_SENSORS_ISL68137 is not set
CONFIG_SENSORS_LM25066=m
CONFIG_SENSORS_LTC2978=m
# CONFIG_SENSORS_LTC3815 is not set
CONFIG_SENSORS_MAX16064=m
# CONFIG_SENSORS_MAX16601 is not set
# CONFIG_SENSORS_MAX20730 is not set
# CONFIG_SENSORS_MAX20751 is not set
# CONFIG_SENSORS_MAX31785 is not set
CONFIG_SENSORS_MAX34440=m
CONFIG_SENSORS_MAX8688=m
# CONFIG_SENSORS_PXE1610 is not set
# CONFIG_SENSORS_TPS40422 is not set
# CONFIG_SENSORS_TPS53679 is not set
CONFIG_SENSORS_UCD9000=m
CONFIG_SENSORS_UCD9200=m
# CONFIG_SENSORS_XDPE122 is not set
CONFIG_SENSORS_ZL6100=m
CONFIG_SENSORS_SHT15=m
CONFIG_SENSORS_SHT21=m
# CONFIG_SENSORS_SHT3x is not set
# CONFIG_SENSORS_SHTC1 is not set
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_EMC6W201=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_SCH56XX_COMMON=m
CONFIG_SENSORS_SCH5627=m
CONFIG_SENSORS_SCH5636=m
# CONFIG_SENSORS_STTS751 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_ADC128D818 is not set
CONFIG_SENSORS_ADS7828=m
# CONFIG_SENSORS_ADS7871 is not set
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_INA209=m
CONFIG_SENSORS_INA2XX=m
# CONFIG_SENSORS_INA3221 is not set
# CONFIG_SENSORS_TC74 is not set
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
# CONFIG_SENSORS_TMP103 is not set
# CONFIG_SENSORS_TMP108 is not set
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
# CONFIG_SENSORS_TMP513 is not set
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
# CONFIG_SENSORS_W83773G is not set
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
# CONFIG_SENSORS_XGENE is not set

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=m CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
# CONFIG_THERMAL_STATISTICS is not set CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0 CONFIG_THERMAL_HWMON=y CONFIG_THERMAL_WRITABLE_TRIPS=y CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set # CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set CONFIG_THERMAL_GOV_FAIR_SHARE=y CONFIG_THERMAL_GOV_STEP_WISE=y CONFIG_THERMAL_GOV_BANG_BANG=y CONFIG_THERMAL_GOV_USER_SPACE=y
# CONFIG_CLOCK_THERMAL is not set
# CONFIG_DEVFREQ_THERMAL is not set
# CONFIG_THERMAL_EMULATION is not set

#
# Intel thermal drivers
#
CONFIG_INTEL_POWERCLAMP=m
CONFIG_X86_PKG_TEMP_THERMAL=m
CONFIG_INTEL_SOC_DTS_IOSF_CORE=m
# CONFIG_INTEL_SOC_DTS_THERMAL is not set

#
# ACPI INT340X thermal drivers
#
CONFIG_INT340X_THERMAL=m
CONFIG_ACPI_THERMAL_REL=m
# CONFIG_INT3406_THERMAL is not set
CONFIG_PROC_THERMAL_MMIO_RAPL=y

# end of ACPI INT340X thermal drivers

# CONFIG_INTEL_PCH_THERMAL is not set

# end of Intel thermal drivers

# CONFIG_GENERIC_ADC_THERMAL is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
CONFIG_WATCHDOG_SYSFS=y

#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_WDAT_WDT=m
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_ZIIRAVE_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
# CONFIG_EBC_C384_WDT is not set
CONFIG_F71808E_WDT=m
CONFIG_SP5100_TCO=m
CONFIG_SBC_FITPC2_WATCHDOG=m
# CONFIG_EUROTECH_WDT is not set
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=y
CONFIG_IE6XX_WDT=m
CONFIG_ITCO_WDT=y
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
CONFIG_NV_TCO=m
# CONFIG_60XX_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC_SCH311X_WDT=m
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_TQMX86_WDT is not set
CONFIG_VIA_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
CONFIG_INTEL_MEI_WDT=m
# CONFIG_NI903X_WDT is not set
# CONFIG_NIC7018_WDT is not set
# CONFIG_MEN_A21_WDT is not set
CONFIG_XEN_WDT=m

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_SSB_DRIVER_GPIO=y
CONFIG_BCMA_POSSIBLE=y
CONFIG_BCMA=m
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
CONFIG_BCMA_HOST_PCI=y
# CONFIG_BCMA_HOST_SOC is not set
CONFIG_BCMA_DRIVER_PCI=y
CONFIG_BCMA_DRIVER_GMAC_CMN=y
CONFIG_BCMA_DRIVER_GPIO=y
# CONFIG_BCMA_DEBUG is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_AS3711 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_BD9571MWV is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_MADERA is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_SPI is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_MC13XXX_SPI is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_MFD_MP2629 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_HTC_I2CPLD is not set
# CONFIG_MFD_INTEL_QUARK_I2C_GPIO is not set
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=m
# CONFIG_INTEL_SOC_PMIC_CHTDC_TI is not set
CONFIG_MFD_INTEL_LPSS=y
CONFIG_MFD_INTEL_LPSS_ACPI=y
CONFIG_MFD_INTEL_LPSS_PCI=y
# CONFIG_MFD_INTEL_PMC_BXT is not set
# CONFIG_MFD_IQS62X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX14577 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6360 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
# CONFIG_EZX_PCAP is not set
CONFIG_MFD_VIPERBOARD=m
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_UCB1400_CORE is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
CONFIG_MFD_SM501=m
CONFIG_MFD_SM501_GPIO=y
# CONFIG_MFD_SKY81452 is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_TI_LMU is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65086 is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TI_LP873X is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TQMX86 is not set
CONFIG_MFD_VX855=m
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_ARIZONA_SPI is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set

# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
CONFIG_RC_CORE=m
CONFIG_RC_MAP=m
# CONFIG_LIRC is not set
CONFIG_RC_DECODERS=y
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_SANYO_DECODER=m
# CONFIG_IR_SHARP_DECODER is not set
CONFIG_IR_MCE_KBD_DECODER=m
# CONFIG_IR_XMP_DECODER is not set
# CONFIG_IR_IMON_DECODER is not set
# CONFIG_IR_RCMM_DECODER is not set
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
# CONFIG_IR_IMON_RAW is not set
CONFIG_IR_MCEUSB=m
CONFIG_IR_ITE_CIR=m
CONFIG_IR_FINTEK=m
CONFIG_IR_NUVOTON=m
CONFIG_IR_REDRAT3=m
CONFIG_IR_STREAMZAP=m
CONFIG_IR_WINBOND_CIR=m
# CONFIG_IR_IGORPLUGUSB is not set
CONFIG_IR_IGUANA=m
CONFIG_IR_TTUSBIR=m
# CONFIG_RC_LOOPBACK is not set
# CONFIG_IR_SERIAL is not set
# CONFIG_IR_SIR is not set
# CONFIG_RC_XBOX_DVD is not set
# CONFIG_MEDIA_CEC_SUPPORT is not set
CONFIG_MEDIA_SUPPORT=m
CONFIG_MEDIA_SUPPORT_FILTER=y
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y

#
# Media device types
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
CONFIG_MEDIA_RADIO_SUPPORT=y
# CONFIG_MEDIA_SDR_SUPPORT is not set
# CONFIG_MEDIA_PLATFORM_SUPPORT is not set
# CONFIG_MEDIA_TEST_SUPPORT is not set

# end of Media device types

CONFIG_VIDEO_DEV=m
CONFIG_MEDIA_CONTROLLER=y
CONFIG_DVB_CORE=m

#
# Video4Linux options
#
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEO_V4L2_I2C=y
# CONFIG_VIDEO_V4L2_SUBDEV_API is not set
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m

# end of Video4Linux options

#
# Media controller options
#
CONFIG_MEDIA_CONTROLLER_DVB=y

# end of Media controller options

#
# Digital TV options
#
# CONFIG_DVB_MMAP is not set
CONFIG_DVB_NET=y
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
# CONFIG_DVB_DEMUX_SECTION_LOSS_LOG is not set
# CONFIG_DVB_ULE_DEBUG is not set

# end of Digital TV options

#
# Media drivers
#

#
# Drivers filtered as selected at 'Filter media drivers' #
CONFIG_TTPCI_EEPROM=m CONFIG_MEDIA_USB_SUPPORT=y

#
# Webcam devices
#
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
# CONFIG_USB_GSPCA_DTCS033 is not set
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_JL2005BCD=m
# CONFIG_USB_GSPCA_KINECT is not set
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
CONFIG_USB_GSPCA_NW80X=m
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SE401=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
# CONFIG_USB_GSPCA_STK1135 is not set
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TOPRO=m
# CONFIG_USB_GSPCA_TOUPTEK is not set
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
CONFIG_USB_GSPCA_VICAM=m
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
# CONFIG_VIDEO_CPIA2 is not set
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
# CONFIG_VIDEO_USBTV is not set

#
# Analog TV USB devices
#
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
# CONFIG_VIDEO_STK1160_COMMON is not set
# CONFIG_VIDEO_GO7007 is not set

#
# Analog/digital TV USB devices
#
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_AU0828_V4L2=y
# CONFIG_VIDEO_AU0828_RC is not set
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m

#
# Digital TV USB devices
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_DIB3000MC=m
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
# CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
# CONFIG_DVB_USB_CXUSB_ANALOG is not set
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_PCTV452E=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_TECHNISAT_USB2=m
CONFIG_DVB_USB_V2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_AF9035=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_AZ6007=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_LME2510=m
CONFIG_DVB_USB_MXL111SF=m
CONFIG_DVB_USB_RTL28XXU=m
# CONFIG_DVB_USB_DVBSKY is not set
# CONFIG_DVB_USB_ZD1301 is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_USB_DRV=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_USB_DEBUG is not set
# CONFIG_DVB_AS102 is not set

#
# Webcam, TV (analog/digital) USB devices
#
CONFIG_VIDEO_EM28XX=m
# CONFIG_VIDEO_EM28XX_V4L2 is not set
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_EM28XX_RC=m
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#
# CONFIG_VIDEO_MEYE is not set
# CONFIG_VIDEO_SOLO6X10 is not set
# CONFIG_VIDEO_TW5864 is not set
# CONFIG_VIDEO_TW68 is not set
# CONFIG_VIDEO_TW686X is not set

#
# Media capture/analog TV support
#
CONFIG_VIDEO_IVTV=m
# CONFIG_VIDEO_IVTV_DEPRECATED_IOCTLS is not set
# CONFIG_VIDEO_IVTV_ALSA is not set
CONFIG_VIDEO_FB_IVTV=m
# CONFIG_VIDEO_FB_IVTV_FORCE_PAT is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DT3155 is not set

#
# Media capture/analog/hybrid TV support
#
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_CX23885=m
CONFIG_MEDIA_ALTERA_CI=m
# CONFIG_VIDEO_CX25821 is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_ENABLE_VP3054=y
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_BT848=m
CONFIG_DVB_BT8XX=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_SAA7164=m

#
# Media digital TV PCI Adapters
#
CONFIG_DVB_AV7110_IR=y
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
# CONFIG_DVB_B2C2_FLEXCOP_PCI_DEBUG is not set
CONFIG_DVB_PLUTO2=m
CONFIG_DVB_DM1105=m
CONFIG_DVB_PT1=m
# CONFIG_DVB_PT3 is not set
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
CONFIG_DVB_NGENE=m
CONFIG_DVB_DDBRIDGE=m
# CONFIG_DVB_DDBRIDGE_MSIENABLE is not set
# CONFIG_DVB_SMIPCIE is not set
# CONFIG_DVB_NETUP_UNIDVB is not set
# CONFIG_VIDEO_IPU3_CIO2 is not set
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_TEA575X=m
# CONFIG_RADIO_SI470X is not set
# CONFIG_RADIO_SI4713 is not set
# CONFIG_USB_MR800 is not set
# CONFIG_USB_DSBR is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_SHARK is not set
# CONFIG_RADIO_SHARK2 is not set
# CONFIG_USB_KEENE is not set
# CONFIG_USB_RAREMONO is not set
# CONFIG_USB_MA901 is not set
# CONFIG_RADIO_TEA5764 is not set
# CONFIG_RADIO_SAA7706H is not set
# CONFIG_RADIO_TEF6862 is not set
# CONFIG_RADIO_WL1273 is not set
CONFIG_MEDIA_COMMON_OPTIONS=y

#

# common driver options
#
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_CYPRESS_FIRMWARE=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_V4L2=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEOBUF2_DMA_SG=m
CONFIG_VIDEOBUF2_DVB=m
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_SMS_SIANO_MDTV=m
CONFIG_SMS_SIANO_RC=y

#
# FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y

# end of Media drivers

CONFIG_MEDIA_HIDE_ANCILLARY_SUBDRV=y

#
# Media ancillary drivers
#
CONFIG_MEDIA_ATTACH=y

#
# IR I2C driver auto-selected by 'Autoselect ancillary drivers' #
CONFIG_VIDEO_IR_I2C=m

#
# audio, video and radio I2C drivers auto-selected by 'Autoselect ancillary drivers' #
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS3308=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m
CONFIG_VIDEO_SAA6588=m
CONFIG_VIDEO_SAA711X=m

#
# Video and audio decoders
#
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_CX25840=m
CONFIG_VIDEO_SAA7127=m
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
CONFIG_VIDEO_SAA6752HS=m
CONFIG_VIDEO_M52790=m

#
# Camera sensor devices
#
# CONFIG_VIDEO_HI556 is not set
# CONFIG_VIDEO_IMX219 is not set
# CONFIG_VIDEO_IMX258 is not set
# CONFIG_VIDEO_IMX274 is not set
# CONFIG_VIDEO_IMX290 is not set
# CONFIG_VIDEO_IMX319 is not set
# CONFIG_VIDEO_IMX355 is not set
# CONFIG_VIDEO_OV2640 is not set
# CONFIG_VIDEO_OV2659 is not set
# CONFIG_VIDEO_OV2680 is not set
# CONFIG_VIDEO_OV2685 is not set
# CONFIG_VIDEO_OV2740 is not set
# CONFIG_VIDEO_OV5647 is not set
# CONFIG_VIDEO_OV6650 is not set
# CONFIG_VIDEO_OV5670 is not set
# CONFIG_VIDEO_OV5675 is not set
# CONFIG_VIDEO_OV5695 is not set
# CONFIG_VIDEO_OV7251 is not set
# CONFIG_VIDEO_OV772X is not set
# CONFIG_VIDEO_OV7640 is not set
# CONFIG_VIDEO_OV7670 is not set
# CONFIG_VIDEO_OV7740 is not set
# CONFIG_VIDEO_OV8856 is not set
# CONFIG_VIDEO_OV9640 is not set
# CONFIG_VIDEO_OV9650 is not set
# CONFIG_VIDEO_OV13858 is not set
# CONFIG_VIDEO_VS6624 is not set
# CONFIG_VIDEO_MT9M001 is not set
# CONFIG_VIDEO_MT9M032 is not set
# CONFIG_VIDEO_MT9M111 is not set
# CONFIG_VIDEO_MT9P031 is not set
# CONFIG_VIDEO_MT9T001 is not set
# CONFIG_VIDEO_MT9T112 is not set
# CONFIG_VIDEO_MT9V011 is not set
# CONFIG_VIDEO_MT9V032 is not set
# CONFIG_VIDEO_MT9V111 is not set
# CONFIG_VIDEO_SR030PC30 is not set
# CONFIG_VIDEO_NOON010PC30 is not set
# CONFIG_VIDEO_M5MOLS is not set
# CONFIG_VIDEO_RJ54N1 is not set
# CONFIG_VIDEO_S5K6AA is not set
# CONFIG_VIDEO_S5K6A3 is not set
# CONFIG_VIDEO_S5K4ECGX is not set
# CONFIG_VIDEO_S5K5BAF is not set
# CONFIG_VIDEO_SMIAPP is not set
# CONFIG_VIDEO_ET8EK8 is not set
# CONFIG_VIDEO_S5C73M3 is not set

# end of Camera sensor devices

#
# Lens drivers
#
# CONFIG_VIDEO_AD5820 is not set
# CONFIG_VIDEO_AK7375 is not set
# CONFIG_VIDEO_DW9714 is not set
# CONFIG_VIDEO_DW9807_VCM is not set

3,813 lines not shown · raw message

Part of this message is not shown here. View the raw message.

maintainer
#4 lore raw
In reply to Mathieu Desnoyers

include/linux/sched.h struct task_struct { · line 694

Quoted 11 lines · Mathieu Desnoyers
include/linux/sched.h +7 -0
@@ -660,6 +685,13 @@ struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
 	unsigned int			cpu;
+#endif
+#ifdef CONFIG_SCHED_PAIR_CPU
+	int				pair_cpu;
+	struct kthread_work		pair_cpu_work;
+	int				pair_cpu_need_worker;
+	int				pair_cpu_worker_active;
+	int				pair_cpu_queued_work;

That's bad layout.

Quoted 19 lines
include/linux/sched.h +0 -0
 #endif
 	unsigned int			wakee_flips;
 	unsigned long			wakee_flip_decay_ts;
kernel/sched/core.c +5 -0
index 8f360326861e..4f1ba8c7d292 100644
@@ -39,6 +39,25 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_overutilized_tp);
 
 DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
 
+#ifdef CONFIG_SCHED_PAIR_CPU
+
+#define MAX_SCHED_PAIR_CPU_WORK_NS	4000000		/* 4 ms */
+
+#define SCHED_PAIR_CPU_CMD_BITMASK	\

kernel/sched/core.c EXPORT_TRACEPOINT_SYMBOL_GPL(sched_overutilized_tp); · line 56

Quoted 10 lines
kernel/sched/core.c +10 -0
+	(SCHED_PAIR_CPU_CMD_SET | SCHED_PAIR_CPU_CMD_CLEAR)
+
+struct pair_cpu {
+	struct kthread_worker *worker;
+	int cpu;				/* protected cpu */
+	int worker_preempted;
+	struct task_struct *running;
+};
+
+static DEFINE_PER_CPU(struct pair_cpu, pair_cpu);

Probably wants to be in rq

Quoted 11 lines
kernel/sched/core.c +4 -0
+static enum cpuhp_state pair_cpu_hp_online;
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 #if defined(CONFIG_SCHED_DEBUG) && defined(CONFIG_JUMP_LABEL)
 /*
  * Debugging: various feature bits
@@ -1692,6 +1711,150 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1723

Quoted 10 lines
kernel/sched/core.c +10 -0
+#ifdef CONFIG_SCHED_PAIR_CPU
+
+static void pair_cpu_preempt_ipi(void *data)
+{
+	trace_printk("pair_cpu_preempt_ipi on cpu %d task %p\n", smp_processor_id(), current);
+	/*
+	 * Order prior userspace memory accesses of local CPU with following
+	 * remote userspace memory accesses.
+	 */
+	smp_mb();

After one reading, I'm still terminally confused what goes where. But I think this is implied by the smp_call_function_single() you're using.

It does:

	csd->func();
	csd_unlock() := smp_store_release()

while the calling CPU does:

	csd_lock_wait() := smp_cond_load_acquire();

Also:

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1724

Quoted 1 line
kernel/sched/core.c +1 -0
+	set_tsk_need_resched(current);

that's set_bit(), which is an atomic operation. But it also isn't sufficient, at the very least it needs set_preempt_need_resched(). Also see resched_curr();

Quoted 10 lines
kernel/sched/core.c +8 -0
+}
 
 
+
+static void pair_cpu_work_func(struct kthread_work *work)
+{
+	struct task_struct *task = container_of(work, struct task_struct,
+						pair_cpu_work);
+	int task_pair_cpu = READ_ONCE(task->pair_cpu);
+	struct pair_cpu *cpum;

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1742

Quoted 10 lines
kernel/sched/core.c +10 -0
+	ktime_t time_begin = ktime_get();
+	bool timeout = false;
+
+	WARN_ON_ONCE(task_pair_cpu < 0);
+	cpum = per_cpu_ptr(&pair_cpu, task_pair_cpu);
+
+	preempt_disable();
+	/* Set worker active for task. */
+	WRITE_ONCE(cpum->worker_preempted, 0);
+	WRITE_ONCE(cpum->running, task);

I'm confused by this, we're the worker, and what is task?

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1748

Quoted 6 lines
kernel/sched/core.c +6 -0
+	/*
+	 * Order prior userspace memory accesses of local CPU with following
+	 * remote userspace memory accesses.
+	 */
+	smp_mb();
+	WRITE_ONCE(task->pair_cpu_worker_active, 1);

Don't we have smp_store_release() for that? Comment also fails to point to pairing barrier.

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1756

Quoted 8 lines
kernel/sched/core.c +8 -0
+	trace_printk("active from cpu %d task %p\n", smp_processor_id(), task);
+	preempt_enable();
+
+	trace_printk("wakeup from cpu %d task %p\n", smp_processor_id(), task);
+	/*
+	 * Wake up target task.
+	 */
+	wake_up_process(task);

That comment is the canonical bad comment. It says abosolutely nothing the code doesn't already say. Comment are supposed to clarify things that are not obvious from the code, like wth task is and where it's supposed to be blocked on. Why are we waking it now?

Quoted 4 lines
kernel/sched/core.c +4 -0
+	/*
+	 * Consume CPU time as long as an associated task is running on another
+	 * CPU.
+	 */

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1771

Quoted 10 lines
kernel/sched/core.c +10 -0
+	while (READ_ONCE(task->pair_cpu_need_worker)
+	       && !READ_ONCE(cpum->worker_preempted)
+	       && task->state != TASK_DEAD) {
+		timeout = ktime_sub(ktime_get(), time_begin) > MAX_SCHED_PAIR_CPU_WORK_NS;
+		if (timeout)
+			break;
+		touch_softlockup_watchdog();
+		cond_resched();
+		cpu_relax();
+	}

Why does it have a timeout? And this is a radically weird approach to testing if @task is currently running. And if we're the worker, we can't be preempted, per definition.

If we're supposed to soak up cycles while @task runs, wth does it allow scheduling in this loop?

Basicaly I think this wants to be something like:

	while (READ_ONCE(task->on_cpu) && READ_ONCE(task->pair_cpu) == cpu)
		cpu_relax();

Nothing more, nothing less.

Quoted 7 lines
kernel/sched/core.c +7 -0
+
+	trace_printk("inactive from cpu %d task %p\n", smp_processor_id(), task);
+	WRITE_ONCE(cpum->running, NULL);
+	WRITE_ONCE(task->pair_cpu_worker_active, 0);
+
+	if (timeout) {
+		/*

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1788

Quoted 10 lines
kernel/sched/core.c +10 -0
+		 * If worker timed out, we need to preempt the associated task with
+		 * an IPI. The IPI may fail if targetting an offline cpu. This implies
+		 * that a preemption of the target task has happened since it ran on
+		 * that cpu.
+		 */
+		int cpu = task_cpu(task);
+
+		trace_printk("worker timeout from cpu %d task %p task_cpu %d\n", smp_processor_id(), task, cpu);
+		if (cpu >= 0)
+			smp_call_function_single(cpu, pair_cpu_preempt_ipi, NULL, 1);

None of this makes sense; task_cpu() will always be >= 0, and if it timed out, it can't be on an offline cpu, because offline CPUs don't run tasks.

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1795

Quoted 7 lines
kernel/sched/core.c +7 -0
+	}
+
+	/*
+	 * Order prior userspace memory accesses of remote CPU with following
+	 * local userspace memory accesses.
+	 */
+	smp_mb();

What?

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1798

Quoted 2 lines
kernel/sched/core.c +2 -0
+
+	put_task_struct(task);

That's an atomic with release semantics.

Quoted 8 lines
kernel/sched/core.c +8 -0
+}
+
+void __sched_pair_cpu_handle_notify_resume(struct ksignal *sig,
+					   struct pt_regs *regs)
+{
+	int task_pair_cpu = READ_ONCE(current->pair_cpu);
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, task_pair_cpu);
+

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1815

Quoted 10 lines
kernel/sched/core.c +10 -0
+	WARN_ON_ONCE(task_pair_cpu < 0);
+	preempt_disable();
+	if (task_pair_cpu == smp_processor_id()) {
+		WRITE_ONCE(current->pair_cpu_need_worker, 0);
+		preempt_enable();
+		if (current->pair_cpu_queued_work) {
+			if (kthread_cancel_work_sync(&current->pair_cpu_work))
+				put_task_struct(current);
+			current->pair_cpu_queued_work = 0;
+		}

Why do we need pair_cpu_queued_work ? Can't we see that from pair_cpu_work state itself?

Quoted 4 lines
kernel/sched/core.c +4 -0
+		trace_printk("notify resume run same cpu for cpu %d from task %p\n", task_pair_cpu,
+		       current);
+		return;
+	}

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1829

Quoted 10 lines
kernel/sched/core.c +10 -0
+	if (READ_ONCE(current->pair_cpu_worker_active)) {
+		preempt_enable();
+		trace_printk("notify resume run diff cpu for cpu %d from task %p\n", task_pair_cpu,
+		       current);
+		/*
+		 *
+		 * Order prior userspace memory accesses of remote CPU with
+		 * following local userspace memory accesses.
+		 */
+		smp_mb();

smp_load_acquire(->pair_cpu_worker_active);

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1838

Quoted 11 lines
kernel/sched/core.c +9 -0
+		return;
+	}
 
 
+	preempt_enable();
+
+	if (current->pair_cpu_queued_work) {
+		if (kthread_cancel_work_sync(&current->pair_cpu_work))
+			put_task_struct(current);
+		current->pair_cpu_queued_work = 0;
+	}

Wait what ? You want it queued here, why waste cycles removing it.

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1851

Quoted 13 lines
kernel/sched/core.c +13 -0
+
+	preempt_disable();
+	set_current_state(TASK_INTERRUPTIBLE);
+	trace_printk("notify resume block for cpu %d from task %p state 0x%lx\n", task_pair_cpu,
+	       current, current->state);
+	WARN_ON_ONCE(current->pair_cpu_worker_active);
+	WRITE_ONCE(current->pair_cpu_need_worker, 1);
+	get_task_struct(current);
+	kthread_init_work(&current->pair_cpu_work, pair_cpu_work_func);
+	kthread_queue_work(cpum->worker, &current->pair_cpu_work);
+	current->pair_cpu_queued_work = 1;
+	preempt_enable();
+	schedule();

Did you consider spurious wakeups? A naked schedule() pretty much always wrong. They need to come in loops and testing variables.

Basically this should be something like:

	rcu_wait_event(.., smp_load_acquire(&current->pair_cpu_worker_active),
			TASK_INTERRUPTIBLE);

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1854

Quoted 3 lines
kernel/sched/core.c +3 -0
+	trace_printk("notify resume unblock for cpu %d from task %p state 0x%lx\n", task_pair_cpu,
+	       current, current->state);
+}

Also, I can't help but think this should've been a task_work.

Quoted 20 lines
kernel/sched/core.c +13 -0
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 #ifdef CONFIG_SCHED_DEBUG
@@ -3181,6 +3344,99 @@ static inline void finish_lock_switch(struct rq *rq)
 # define finish_arch_post_lock_switch()	do { } while (0)
 #endif
 
+#ifdef CONFIG_SCHED_PAIR_CPU
+
+/*
+ * If we preempt the cpu mutex worker, we need to IPI the CPU
+ * running the thread currently associated to it before scheduling
+ * other tasks.
+ *
+ * This only targets pair_cpu worker for online cpus.
+ */
+static void pair_cpu_finish_switch_worker(struct task_struct *prev)

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3366

Quoted 10 lines
kernel/sched/core.c +10 -0
+{
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, smp_processor_id());
+	struct task_struct *running_task;
+	int cpu;
+
+	if (!cpum->worker || prev != cpum->worker->task)
+		return;
+	running_task = READ_ONCE(cpum->running);
+	if (!running_task)
+		return;

You're doing this for _every_ context switch, the above is unconditional new lines touched and a big no-no. Also see the performance regression 0day gifted you.

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3369

Quoted 3 lines
kernel/sched/core.c +3 -0
+	WRITE_ONCE(cpum->worker_preempted, 1);
+	WRITE_ONCE(running_task->pair_cpu_worker_active, 0);
+	WRITE_ONCE(cpum->running, NULL);

I can't help but feel you have _way_ too much duplicate state.

Quoted 6 lines
kernel/sched/core.c +6 -0
+	/*
+	 * If worker was preempted, we need to preempt the associated task with
+	 * an IPI. The IPI may fail if targetting an offline cpu. This implies
+	 * that a preemption of the target task has happened since it ran on
+	 * that cpu.
+	 */

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3386

Quoted 10 lines
kernel/sched/core.c +10 -0
+	cpu = task_cpu(running_task);
+
+	trace_printk("worker preempted from cpu %d task %p task_cpu %d\n", smp_processor_id(), running_task, cpu);
+	if (cpu >= 0) {
+		smp_call_function_single(cpu, pair_cpu_preempt_ipi, NULL, 1);
+		/*
+		 * Order prior userspace memory accesses of remote CPU with
+		 * following local userspace memory accesses.
+		 */
+		smp_mb();

Already guaranteed by smp_call_function_single(), as explained earlier.

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3397

Quoted 11 lines
kernel/sched/core.c +11 -0
+	}
+}
+
+
+static void pair_cpu_remote_mb(void *data)
+{
+       /*
+        * Order prior userspace memory accesses of remote CPU with following
+        * local userspace memory accesses.
+        */
+       smp_mb();

I think you can do without, see callsite.

Quoted 4 lines
kernel/sched/core.c +4 -0
+}
+
+/*
+ * If we preempt a task currently associated with a cpu mutex worker,

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3411

Quoted 10 lines
kernel/sched/core.c +10 -0
+ * we need to tell the worker to stop using cpu time.
+ */
+static void pair_cpu_finish_switch_task(struct task_struct *prev, long prev_state)
+{
+	int prev_pair_cpu;
+
+	prev_pair_cpu = READ_ONCE(prev->pair_cpu);
+	if (prev_pair_cpu < 0 || !READ_ONCE(prev->pair_cpu_need_worker)
+	    || !READ_ONCE(prev->pair_cpu_worker_active))
+		return;

Again, you're doing this every context switch, touching memory we didn't used to touch.

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1723

Quoted 5 lines
kernel/sched/core.c +5 -0
+	/*
+	* Order prior userspace memory accesses of local CPU with following
+	* remote userspace memory accesses.
+	*/
+	smp_mb();

We're at the end of schedule; there is no way to get here without already having done one of these since userspace.

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3421

Quoted 5 lines
kernel/sched/core.c +5 -0
+	/*
+	* IPI may fail if CPU is offlined, in which case the memory barrier
+	* before the worker completes will suffice.
+	*/
+	smp_call_function_single(prev_pair_cpu, pair_cpu_remote_mb, NULL, 1);

per the .wait=1 you already have a remote store-release and local load-acquire chain.

Quoted 22 lines
kernel/sched/core.c +18 -0
+	WRITE_ONCE(prev->pair_cpu_worker_active, 0);
+	WRITE_ONCE(prev->pair_cpu_need_worker, 0);
+}
+
+static void pair_cpu_finish_switch(struct task_struct *prev, long prev_state)
+{
+	pair_cpu_finish_switch_worker(prev);
+	pair_cpu_finish_switch_task(prev, prev_state);
+}
+
+#else /* CONFIG_SCHED_PAIR_CPU */
+
+static void pair_cpu_finish_switch(struct task_struct *prev, long prev_state)
+{
+}
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 /**
  * prepare_task_switch - prepare to switch tasks
  * @rq: the runqueue preparing to switch

kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct *prev) · line 3528

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -3269,6 +3525,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	finish_lock_switch(rq);
 	finish_arch_post_lock_switch();
 	kcov_finish_switch(current);
+	pair_cpu_finish_switch(prev, prev_state);

So you're adding a bunch of cacheline misses here

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4383

Quoted 8 lines
kernel/sched/core.c +1 -0
 	fire_sched_in_preempt_notifiers(current);
 	/*
@@ -4123,6 +4380,7 @@ static void __sched notrace __schedule(bool preempt)
 	next = pick_next_task(rq, prev, &rf);
 	clear_tsk_need_resched(prev);
 	clear_preempt_need_resched();
+	sched_pair_cpu_preempt(prev);

And one here.

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4386

Quoted 3 lines
kernel/sched/core.c +0 -0
 	if (likely(prev != next)) {
 		rq->nr_switches++;

I think if you use a task instead of a kthread_work things become much simpler. Regardless, you have too much state which mightily confuses things.

As with all the other approaches, I don't love this.. All of this complexity is because sodding hotplug -- something that should really never happen.

So you're proposing adding code to incredibly hot paths to compensate for the absolute slowest path in the entire kernel, that can't be right.

Things I would hate less:

- making open-files of /sys/devices/system/cpu/cpuN/online pin the hotplug state (yuck! :-)

- make fcntl(F_*SIG) work on /sys/devices/system/cpu/cpuN/online such that we receive a signal when the state changes, and before userspace starts. The signal triggers rseq restart.

- sched_setaffinity2() that includes a flags argument.

with this it might be possible to expose a flag that allows userspace to affine to offline CPUs, with the promise that when the CPU goes online it will migrate the task before it starts running userspace. This migration triggers rseq restart.

I think the second probably has the best complexity:yuck ratio.

#5 lore raw
Quoted 15 lines · ----- On Jun 24, 2020, at 8:11 AM, Peter Zijlstra [email protected] wrote:

include/linux/sched.h struct task_struct { · line 694

Quoted 11 lines · Mathieu Desnoyers
include/linux/sched.h +7 -0
@@ -660,6 +685,13 @@ struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
 	unsigned int			cpu;
+#endif
+#ifdef CONFIG_SCHED_PAIR_CPU
+	int				pair_cpu;
+	struct kthread_work		pair_cpu_work;
+	int				pair_cpu_need_worker;
+	int				pair_cpu_worker_active;
+	int				pair_cpu_queued_work;

That's bad layout.

Those should probably be moved to a bitmask instead, and also moved further down in the task_struct so we don't take room which is usually taken by scheduler fast-path cachelines.

But until now it was enough to show that it can at least survive testing. :)

Quoted 24 lines
Quoted 24 lines
include/linux/sched.h +0 -0
 #endif
 	unsigned int			wakee_flips;
 	unsigned long			wakee_flip_decay_ts;
kernel/sched/core.c +9 -0
index 8f360326861e..4f1ba8c7d292 100644
@@ -39,6 +39,25 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_overutilized_tp);
 
 DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
 
+#ifdef CONFIG_SCHED_PAIR_CPU
+
+#define MAX_SCHED_PAIR_CPU_WORK_NS	4000000		/* 4 ms */
+
+#define SCHED_PAIR_CPU_CMD_BITMASK	\
+	(SCHED_PAIR_CPU_CMD_SET | SCHED_PAIR_CPU_CMD_CLEAR)
+
+struct pair_cpu {
+	struct kthread_worker *worker;
Quoted 8 lines

kernel/sched/core.c EXPORT_TRACEPOINT_SYMBOL_GPL(sched_overutilized_tp); · line 56

Quoted 6 lines
kernel/sched/core.c +6 -0
+	int cpu;				/* protected cpu */
+	int worker_preempted;
+	struct task_struct *running;
+};
+
+static DEFINE_PER_CPU(struct pair_cpu, pair_cpu);

Probably wants to be in rq

Good point! Will move it there.

Quoted 28 lines
Quoted 14 lines
kernel/sched/core.c +5 -0
+static enum cpuhp_state pair_cpu_hp_online;
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 #if defined(CONFIG_SCHED_DEBUG) && defined(CONFIG_JUMP_LABEL)
 /*
  * Debugging: various feature bits
@@ -1692,6 +1711,150 @@ int set_cpus_allowed_ptr(struct task_struct *p, const
 struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
+#ifdef CONFIG_SCHED_PAIR_CPU

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const · line 1725

Quoted 10 lines
kernel/sched/core.c +9 -0
+
+static void pair_cpu_preempt_ipi(void *data)
+{
+	trace_printk("pair_cpu_preempt_ipi on cpu %d task %p\n", smp_processor_id(),
 current);
+	/*
+	 * Order prior userspace memory accesses of local CPU with following
+	 * remote userspace memory accesses.
+	 */
+	smp_mb();

After one reading, I'm still terminally confused what goes where. But I think this is implied by the smp_call_function_single() you're using.

Quoted 8 lines

It does:

	csd->func();
	csd_unlock() := smp_store_release()

while the calling CPU does:

	csd_lock_wait() := smp_cond_load_acquire();

Good point! The smp_mb is redundant with the needed acquire/release semantic already provided by csd in this case.

Quoted 9 lines

Also:

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1724

Quoted 1 line
kernel/sched/core.c +1 -0
+	set_tsk_need_resched(current);

that's set_bit(), which is an atomic operation. But it also isn't sufficient, at the very least it needs set_preempt_need_resched(). Also see resched_curr();

Indeed, set_preempt_need_resched is needed. I don't think I need to grab the runqueue lock of the target CPU however, because any task migration would guarantee that the task has been preempted.

Quoted 15 lines
Quoted 15 lines
kernel/sched/core.c +12 -0
+}
 
 
+
+static void pair_cpu_work_func(struct kthread_work *work)
+{
+	struct task_struct *task = container_of(work, struct task_struct,
+						pair_cpu_work);
+	int task_pair_cpu = READ_ONCE(task->pair_cpu);
+	struct pair_cpu *cpum;
+	ktime_t time_begin = ktime_get();
+	bool timeout = false;
+
+	WARN_ON_ONCE(task_pair_cpu < 0);
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1742

Quoted 6 lines
kernel/sched/core.c +6 -0
+	cpum = per_cpu_ptr(&pair_cpu, task_pair_cpu);
+
+	preempt_disable();
+	/* Set worker active for task. */
+	WRITE_ONCE(cpum->worker_preempted, 0);
+	WRITE_ONCE(cpum->running, task);

I'm confused by this, we're the worker, and what is task?

The worker is the kernel worker thread affined to the CPU which "owns" the per-cpu data.

The "task" is the task which invokes sched_pair_cpu() to be paired with that CPU.

Quoted 10 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1748

Quoted 7 lines
kernel/sched/core.c +6 -0
+	/*
+	 * Order prior userspace memory accesses of local CPU with following
+	 * remote userspace memory accesses.
+	 */
+	smp_mb();
+	WRITE_ONCE(task->pair_cpu_worker_active, 1);

Don't we have smp_store_release() for that? Comment also fails to point to pairing barrier.

Indeed, I'll change those explicit smp_mb() into acquire/release of:

pair_cpu_need_worker: acquired by worker thread, released by paired task.
pair_cpu_worker_active: acquired by paired task, released by worker thread.

Quoted 14 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1756

Quoted 9 lines
kernel/sched/core.c +8 -0
+	trace_printk("active from cpu %d task %p\n", smp_processor_id(), task);
+	preempt_enable();
+
+	trace_printk("wakeup from cpu %d task %p\n", smp_processor_id(), task);
+	/*
+	 * Wake up target task.
+	 */
+	wake_up_process(task);

That comment is the canonical bad comment. It says abosolutely nothing the code doesn't already say. Comment are supposed to clarify things that are not obvious from the code, like wth task is and where it's supposed to be blocked on. Why are we waking it now?

Changing for:

        /*
         * The paired task has queued this kthread work and is blocked awaiting
         * for this thread to set "pair_cpu_worker_active" and awaken it.
         */
        wake_up_process(task);
Quoted 17 lines
Quoted 5 lines
kernel/sched/core.c +4 -0
+	/*
+	 * Consume CPU time as long as an associated task is running on another
+	 * CPU.
+	 */

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1771

Quoted 10 lines
kernel/sched/core.c +10 -0
+	while (READ_ONCE(task->pair_cpu_need_worker)
+	       && !READ_ONCE(cpum->worker_preempted)
+	       && task->state != TASK_DEAD) {
+		timeout = ktime_sub(ktime_get(), time_begin) > MAX_SCHED_PAIR_CPU_WORK_NS;
+		if (timeout)
+			break;
+		touch_softlockup_watchdog();
+		cond_resched();
+		cpu_relax();
+	}

Why does it have a timeout?

The timeout is used to handle case where the kernel worker thread is handling the per-cpu data of an offline CPU. In that peculiar case, the worker thread will be running on a CPU which is not the offline CPU, so no preemption will set the "worker_preempted" flag in the struct pair_cpu as long as the CPU is offline. The timeout ensures that a single worker task does not stay arbitrarily long as "current", and gets kicked away to leave room for other tasks which may also need to access that offlined cpu's per-cpu data.

Quoted 3 lines

And this is a radically weird approach to
testing if @task is currently running. And if we're the worker, we can't be preempted, per definition.

Note that the "worker_preempted" test is really testing whether the _worker_ has been preempted.

If we are the worker, we _can_ and _should_ be preempted if something else needs to be scheduled. We don't want to prevent more important tasks to run.

This implies that whatever the task calling sched_pair_cpu set/clear does between set and clear to touch per-cpu data needs to be done with restartable sequences, so the critical section is aborted if the task is preempted. Same goes if the kworker thread is preempted: it preempts the paired task with an IPI, which immediately prevents its user-space to touch the per-cpu data until the kworker is scheduled back and picks the eventually re-queued work for another round.

Quoted 10 lines

If we're supposed to soak up cycles while @task runs, wth does it allow scheduling in this loop?

Basicaly I think this wants to be something like:

	while (READ_ONCE(task->on_cpu) && READ_ONCE(task->pair_cpu) == cpu)
		cpu_relax();

Nothing more, nothing less.

The semantic of what you propose is:

- Stop running as soon as @task is not running,
- Stop running as soon as @task is not paired with the cpu serviced by this worker thread.

The semantic I need is:

- Stop running as soon as @task does not need the worker thread,
- Stop running as soon as the worker thread is preempted,
- Stop running as soon as @task has exited,
- Stop running as soon as timeout is reached.

Quoted 15 lines
Quoted 15 lines
kernel/sched/core.c +14 -0
+
+	trace_printk("inactive from cpu %d task %p\n", smp_processor_id(), task);
+	WRITE_ONCE(cpum->running, NULL);
+	WRITE_ONCE(task->pair_cpu_worker_active, 0);
+
+	if (timeout) {
+		/*
+		 * If worker timed out, we need to preempt the associated task with
+		 * an IPI. The IPI may fail if targetting an offline cpu. This implies
+		 * that a preemption of the target task has happened since it ran on
+		 * that cpu.
+		 */
+		int cpu = task_cpu(task);
+
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1788

Quoted 4 lines
kernel/sched/core.c +1 -0
+		trace_printk("worker timeout from cpu %d task %p task_cpu %d\n",

smp_processor_id(), task, cpu);

kernel/sched/core.c +2 -0
+		if (cpu >= 0)
+			smp_call_function_single(cpu, pair_cpu_preempt_ipi, NULL, 1);

None of this makes sense; task_cpu() will always be >= 0, and if it timed out, it can't be on an offline cpu, because offline CPUs don't run tasks.

There are three scenarios to consider here:

1) The kernel worker thread is handling a CPU which is offline: This explains the need for a "timeout", as discussed earlier.

2) The paired task was running on @cpu when task_cpu() is issued, but the CPU is concurrently going offline before we send the IPI: As the comment explains, the preemption of @task is required in order to allow its CPU to go offline makes the IPI redundant.

3) The paired task is blocked on @cpu in parallel with the timeout being reached by the worker thread: AFAIU this would cause cpu < 0. Again in this case, the task has been preempted, which means our IPI would be redundant.

Quoted 11 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1795

Quoted 8 lines
kernel/sched/core.c +7 -0
+	}
+
+	/*
+	 * Order prior userspace memory accesses of remote CPU with following
+	 * local userspace memory accesses.
+	 */
+	smp_mb();

What?

Quoted 4 lines

kernel/sched/core.c

Quoted 2 lines
kernel/sched/core.c +2 -0
+
+	put_task_struct(task);

That's an atomic with release semantics.

Good point.

Although the ordering is really provided by the following for each scenario:

- pair_cpu_need_worker=0: I'm adding a smp_load_acquire() in the worker thread loop paired with smp_store_release() when the paired thread sets it to 0.
- timeout reached: the required smp_call_function_single to IPI the paired thread provides the acquire/release semantic.
- worker_preempted: pair_cpu_finish_switch_worker performs the IPI to the paired thread, which provides acquire/release semantic.
- task DEAD: there is nothing to order against anymore on the remote cpu.

Quoted 14 lines
Quoted 14 lines
kernel/sched/core.c +13 -0
+}
+
+void __sched_pair_cpu_handle_notify_resume(struct ksignal *sig,
+					   struct pt_regs *regs)
+{
+	int task_pair_cpu = READ_ONCE(current->pair_cpu);
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, task_pair_cpu);
+
+	WARN_ON_ONCE(task_pair_cpu < 0);
+	preempt_disable();
+	if (task_pair_cpu == smp_processor_id()) {
+		WRITE_ONCE(current->pair_cpu_need_worker, 0);
+		preempt_enable();
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1815

Quoted 5 lines
kernel/sched/core.c +5 -0
+		if (current->pair_cpu_queued_work) {
+			if (kthread_cancel_work_sync(&current->pair_cpu_work))
+				put_task_struct(current);
+			current->pair_cpu_queued_work = 0;
+		}

Why do we need pair_cpu_queued_work ? Can't we see that from pair_cpu_work state itself?

Good point, I think I'll need the following changes to make it work:

- Initialize pair_cpu_work in sched_pair_cpu set, just before setting current->pair_cpu to a value >= 0. It could alternatively be done on every task creation, but I don't want to slow down a relatively common code path for the sake of a slow path (sched_pair_cpu).

AFAIU doing so would ensure we can issue kthread_cancel_work_sync() on the initialized but non-queued work.

Quoted 11 lines
Quoted 11 lines
kernel/sched/core.c +1 -0
+		trace_printk("notify resume run same cpu for cpu %d from task %p\n",

task_pair_cpu,

kernel/sched/core.c +6 -0
+		       current);
+		return;
+	}
+	if (READ_ONCE(current->pair_cpu_worker_active)) {
+		preempt_enable();
+		trace_printk("notify resume run diff cpu for cpu %d from task %p\n",

task_pair_cpu,

kernel/sched/core.c +1 -0
+		       current);
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1829

Quoted 6 lines
kernel/sched/core.c +6 -0
+		/*
+		 *
+		 * Order prior userspace memory accesses of remote CPU with
+		 * following local userspace memory accesses.
+		 */
+		smp_mb();

smp_load_acquire(->pair_cpu_worker_active);

Done.

Quoted 5 lines
Quoted 3 lines
kernel/sched/core.c +2 -0
+		return;
+	}
Quoted 9 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1815

Quoted 7 lines
kernel/sched/core.c +7 -0
+	preempt_enable();
+
+	if (current->pair_cpu_queued_work) {
+		if (kthread_cancel_work_sync(&current->pair_cpu_work))
+			put_task_struct(current);
+		current->pair_cpu_queued_work = 0;
+	}

Wait what ? You want it queued here, why waste cycles removing it.

I now notice that kthread_queue_work returns whether the work is already pending. That's indeed all I need to check to know whether I need to get a reference to the task struct. I'll remove this unneeded cancel work by using that scheme.

Quoted 19 lines
Quoted 6 lines
kernel/sched/core.c +4 -0
+
+	preempt_disable();
+	set_current_state(TASK_INTERRUPTIBLE);
+	trace_printk("notify resume block for cpu %d from task %p state 0x%lx\n",

task_pair_cpu,

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1851

Quoted 9 lines
kernel/sched/core.c +9 -0
+	       current, current->state);
+	WARN_ON_ONCE(current->pair_cpu_worker_active);
+	WRITE_ONCE(current->pair_cpu_need_worker, 1);
+	get_task_struct(current);
+	kthread_init_work(&current->pair_cpu_work, pair_cpu_work_func);
+	kthread_queue_work(cpum->worker, &current->pair_cpu_work);
+	current->pair_cpu_queued_work = 1;
+	preempt_enable();
+	schedule();

Did you consider spurious wakeups? A naked schedule() pretty much always wrong. They need to come in loops and testing variables.

Quoted 4 lines

Basically this should be something like:

	rcu_wait_event(.., smp_load_acquire(&current->pair_cpu_worker_active),
			TASK_INTERRUPTIBLE);

This schedule() is already in a loop: __sched_pair_cpu_handle_notify_resume() is called from exit_to_usermode_loop().

The state which gets tested on each loop is:

- whether the current task is now running on the paired cpu (task_pair_cpu == smp_processor_id())
- whether current->pair_cpu_worker_active == 1
- else, set current task state as TASK_INTERRUPTIBLE, indicate that it needs the worker thread, and queue work before calling schedule.

If a spurious wakeup happens, this will cause the entire function __sched_pair_cpu_handle_notify_resume to be called again by exit_to_usermode_loop because our call to schedule() will have set the _TIF_NOTIFY_RESUME thread flag (see sched_pair_cpu_preempt() invoked very early in schedule).

I suspect I should be more clever about sched_pair_cpu_preempt(). Rather than placing it right in middle of an extremely hot path in the scheduler, I should rather do:

- Set _TIF_NOTIFY_RESUME flag for current->pair_cpu >= 0 in finish task switch,
- Also set it right after the call to schedule() in __sched_pair_cpu_handle_notify_resume,
- Also set it in pair_cpu_preempt_ipi().

Quoted 7 lines

kernel/sched/core.c

Quoted 5 lines
kernel/sched/core.c +1 -0
+	trace_printk("notify resume unblock for cpu %d from task %p state 0x%lx\n",

task_pair_cpu,

kernel/sched/core.c +2 -0
+	       current, current->state);
+}

Also, I can't help but think this should've been a task_work.

"queuing" that work is needed from IPI context and from finish task switch. As long as we can queue task_work from atomic context, we could indeed use that instead. The struct callback_head would sit in the task struct.

Doing that change means that I cannot rely on the return to usermode loop to call this again after schedule, so I should put that in a loop myself.

Quoted 26 lines
Quoted 26 lines
kernel/sched/core.c +18 -0
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 #ifdef CONFIG_SCHED_DEBUG
@@ -3181,6 +3344,99 @@ static inline void finish_lock_switch(struct rq *rq)
 # define finish_arch_post_lock_switch()	do { } while (0)
 #endif
 
+#ifdef CONFIG_SCHED_PAIR_CPU
+
+/*
+ * If we preempt the cpu mutex worker, we need to IPI the CPU
+ * running the thread currently associated to it before scheduling
+ * other tasks.
+ *
+ * This only targets pair_cpu worker for online cpus.
+ */
+static void pair_cpu_finish_switch_worker(struct task_struct *prev)
+{
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, smp_processor_id());
+	struct task_struct *running_task;
+	int cpu;
+
Quoted 8 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3366

Quoted 5 lines
kernel/sched/core.c +5 -0
+	if (!cpum->worker || prev != cpum->worker->task)
+		return;
+	running_task = READ_ONCE(cpum->running);
+	if (!running_task)
+		return;

You're doing this for _every_ context switch, the above is unconditional new lines touched and a big no-no.

The only state I need to check here is whether the scheduler is preempting a thread which is a pair-cpu kworker thread. This could be a simple flag within an already cache-hot cacheline in the task struct. The rest then becomes a slow-path.

Would that solve your performance concern ?

Quoted 1 line

Also see the performance regression 0day gifted you.

Yes, I've seen it. I suspect it's because of the location where I added the new pair_cpu fields in the task struct. I think I'm kicking out cache-hot fields into additional cache lines. AFAIU moving my pair_cpu fields further down in the task struct should solve this.

Quoted 6 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3369

Quoted 4 lines
kernel/sched/core.c +3 -0
+	WRITE_ONCE(cpum->worker_preempted, 1);
+	WRITE_ONCE(running_task->pair_cpu_worker_active, 0);
+	WRITE_ONCE(cpum->running, NULL);

I can't help but feel you have _way_ too much duplicate state.

Let's do one round of addressing your feedback and respinning the patch, then it will be easier to look into shrinking duplicate state.

Quoted 12 lines
Quoted 12 lines
kernel/sched/core.c +10 -0
+	/*
+	 * If worker was preempted, we need to preempt the associated task with
+	 * an IPI. The IPI may fail if targetting an offline cpu. This implies
+	 * that a preemption of the target task has happened since it ran on
+	 * that cpu.
+	 */
+	cpu = task_cpu(running_task);
+
+	trace_printk("worker preempted from cpu %d task %p task_cpu %d\n",
 smp_processor_id(), running_task, cpu);
+	if (cpu >= 0) {
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1829

Quoted 6 lines
kernel/sched/core.c +6 -0
+		smp_call_function_single(cpu, pair_cpu_preempt_ipi, NULL, 1);
+		/*
+		 * Order prior userspace memory accesses of remote CPU with
+		 * following local userspace memory accesses.
+		 */
+		smp_mb();

Already guaranteed by smp_call_function_single(), as explained earlier.

Yep. Fixed.

Quoted 14 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3397

Quoted 12 lines
kernel/sched/core.c +11 -0
+	}
+}
+
+
+static void pair_cpu_remote_mb(void *data)
+{
+       /*
+        * Order prior userspace memory accesses of remote CPU with following
+        * local userspace memory accesses.
+        */
+       smp_mb();

I think you can do without, see callsite.

I do not think csd provides the right ordering in this case: the acquire/release chain we get from .wait=1 pairs release on the ipi handler with acquire on the caller. What we need here is to pair an acquire on the ipi handler with a release on the caller, which is the opposite.

I clarified this comment:

+        * Order prior userspace memory accesses of caller CPU with following
+        * userspace memory accesses on the CPU running this handler. This is
+        * paired with a memory barrier at the beginning of scheduling.
Quoted 11 lines
Quoted 11 lines
kernel/sched/core.c +9 -0
+}
+
+/*
+ * If we preempt a task currently associated with a cpu mutex worker,
+ * we need to tell the worker to stop using cpu time.
+ */
+static void pair_cpu_finish_switch_task(struct task_struct *prev, long
 prev_state)
+{
+	int prev_pair_cpu;
Quoted 8 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3411

Quoted 5 lines
kernel/sched/core.c +5 -0
+
+	prev_pair_cpu = READ_ONCE(prev->pair_cpu);
+	if (prev_pair_cpu < 0 || !READ_ONCE(prev->pair_cpu_need_worker)
+	    || !READ_ONCE(prev->pair_cpu_worker_active))
+		return;

Again, you're doing this every context switch, touching memory we didn't used to touch.

Same argument about placing this in a single "hot path" flag in the task struct. Would that address your performance concerns ?

Quoted 9 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1723

Quoted 6 lines
kernel/sched/core.c +5 -0
+	/*
+	* Order prior userspace memory accesses of local CPU with following
+	* remote userspace memory accesses.
+	*/
+	smp_mb();

We're at the end of schedule; there is no way to get here without already having done one of these since userspace.

Good point, so this memory barrier can go away, but not the one within pair_cpu_remote_mb.

Quoted 9 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3421

Quoted 6 lines
kernel/sched/core.c +5 -0
+	/*
+	* IPI may fail if CPU is offlined, in which case the memory barrier
+	* before the worker completes will suffice.
+	*/
+	smp_call_function_single(prev_pair_cpu, pair_cpu_remote_mb, NULL, 1);

per the .wait=1 you already have a remote store-release and local load-acquire chain.

As explained above, we need the opposite acquire/release order in this case.

Quoted 22 lines
Quoted 22 lines
kernel/sched/core.c +18 -0
+	WRITE_ONCE(prev->pair_cpu_worker_active, 0);
+	WRITE_ONCE(prev->pair_cpu_need_worker, 0);
+}
+
+static void pair_cpu_finish_switch(struct task_struct *prev, long prev_state)
+{
+	pair_cpu_finish_switch_worker(prev);
+	pair_cpu_finish_switch_task(prev, prev_state);
+}
+
+#else /* CONFIG_SCHED_PAIR_CPU */
+
+static void pair_cpu_finish_switch(struct task_struct *prev, long prev_state)
+{
+}
+
+#endif /* CONFIG_SCHED_PAIR_CPU */
+
 /**
  * prepare_task_switch - prepare to switch tasks
  * @rq: the runqueue preparing to switch
Quoted 8 lines

kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct · line 3529

Quoted 6 lines
kernel/sched/core.c +1 -0
@@ -3269,6 +3525,7 @@ static struct rq *finish_task_switch(struct task_struct
 *prev)
 	finish_lock_switch(rq);
 	finish_arch_post_lock_switch();
 	kcov_finish_switch(current);
+	pair_cpu_finish_switch(prev, prev_state);

So you're adding a bunch of cacheline misses here

It could be turned into a task struct flag test instead.

Quoted 11 lines

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4383

Quoted 9 lines
kernel/sched/core.c +1 -0
 	fire_sched_in_preempt_notifiers(current);
 	/*
@@ -4123,6 +4380,7 @@ static void __sched notrace __schedule(bool preempt)
 	next = pick_next_task(rq, prev, &rf);
 	clear_tsk_need_resched(prev);
 	clear_preempt_need_resched();
+	sched_pair_cpu_preempt(prev);

And one here.

This one can entirely go away as discussed further up.

Quoted 9 lines

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4386

Quoted 4 lines
kernel/sched/core.c +0 -0
 	if (likely(prev != next)) {
 		rq->nr_switches++;

I think if you use a task instead of a kthread_work things become much simpler. Regardless, you have too much state which mightily confuses things.

Do you mean "use a task_work" or really "use a task" ?

AFAIU task_work would only replace the custom return to usermode handler I implemented.

I used a kthread_work to eliminate the need to re-implement the work handling logic. What would be the benefit of using a task rather than kthread_work here ?

Quoted 3 lines

As with all the other approaches, I don't love this.. All of this complexity is because sodding hotplug -- something that should really never happen.

Actually, it's not only hotplug. It's hotplug _and_ cgroup cpusets.

Quoted 3 lines

So you're proposing adding code to incredibly hot paths to compensate for the absolute slowest path in the entire kernel, that can't be right.

As I said above, I'm pretty sure we could optimize the addition to those hot paths to almost nothing.

Quoted 4 lines

Things I would hate less:

- making open-files of /sys/devices/system/cpu/cpuN/online pin the hotplug state (yuck! :-)

I don't see this being available to non-root users, nor to containerized environments. Also, it would not handle cgroup cpusets.

Quoted 3 lines

- make fcntl(F_*SIG) work on /sys/devices/system/cpu/cpuN/online such that we receive a signal when the state changes, and before userspace starts. The signal triggers rseq restart.

Just knowing a CPU is offline is not enough. We then need to be able to synchronize concurrent updates to that per-CPU data structure while the CPU is offline, in a way which is also safe against that CPU coming back online again.

Also it does not solve the issues associated with cgroup cpusets, where a CPU may very well be online, but not part of our thread's allowed cpu mask.

Quoted 7 lines

- sched_setaffinity2() that includes a flags argument.

with this it might be possible to expose a flag that allows userspace to affine to offline CPUs, with the promise that when the CPU goes online it will migrate the task before it starts running userspace. This migration triggers rseq restart.

This is very similar to the earlier "pin_on_cpu" system call I proposed. Note that it would also need to serialize execution of all threads which are affined to that offline cpu, so concurrent updates to an offlined cpu per-cpu data don't corrupt each other.

While we can make the hotplug case work, the main issue remaining with that approach is interaction with cgroup cpusets. What should we do when migration to the requested CPU is disallowed by cgroups ? Note that this cpuset change can be triggered by an external "manager" process concurrently.

We do not want to override the affinity restricted by cgroups because we don't want to hurt performance characteristics of another partition of the system.

The sched_pair_cpu approach has the benefit of allowing us to touch per-cpu data of a given CPU without requiring to run on that CPU, which ensures that we do not thrash the cpu cache of cpus on which a thread is not allowed to run. It takes care of issues caused by both cgroup cpusets and cpu hotplug.

Quoted 2 lines

I think the second probably has the best complexity:yuck ratio.

Unfortunately the cgroup cpuset issues are not taken into account by the 3 approaches you propose above. Those added constraints are what motivated me to implement the sched_pair_cpu system call.

Thanks Peter for the thorough review,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#6 lore raw
Quoted 9 lines · Mathieu Desnoyers
Quoted 5 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1724

Quoted 1 line
kernel/sched/core.c +1 -0
+	set_tsk_need_resched(current);

that's set_bit(), which is an atomic operation. But it also isn't sufficient, at the very least it needs set_preempt_need_resched(). Also see resched_curr();

Indeed, set_preempt_need_resched is needed. I don't think I need to grab the runqueue lock of the target CPU however, because any task migration would guarantee that the task has been preempted.

Correct, you can mark a local reschedule without rq->lock.

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
kernel/sched/core.c +8 -0
+	/*
+	 * Consume CPU time as long as an associated task is running on another
+	 * CPU.
+	 */
+	while (READ_ONCE(task->pair_cpu_need_worker)
+	       && !READ_ONCE(cpum->worker_preempted)
+	       && task->state != TASK_DEAD) {
+		timeout = ktime_sub(ktime_get(), time_begin) > MAX_SCHED_PAIR_CPU_WORK_NS;
Quoted 8 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1771

Quoted 6 lines
kernel/sched/core.c +6 -0
+		if (timeout)
+			break;
+		touch_softlockup_watchdog();
+		cond_resched();
+		cpu_relax();
+	}

Why does it have a timeout?

Quoted 7 lines

The timeout is used to handle case where the kernel worker thread is handling the per-cpu data of an offline CPU. In that peculiar case, the worker thread will be running on a CPU which is not the offline CPU, so no preemption will set the "worker_preempted" flag in the struct pair_cpu as long as the CPU is offline. The timeout ensures that a single worker task does not stay arbitrarily long as "current", and gets kicked away to leave room for other tasks which may also need to access that offlined cpu's per-cpu data.

But if the CPU is offline, you don't actually need to do anything. So that seems unfortunate.

Quoted 6 lines
Quoted 3 lines

And this is a radically weird approach to
testing if @task is currently running. And if we're the worker, we can't be preempted, per definition.

Note that the "worker_preempted" test is really testing whether the _worker_ has been preempted.

But this is the worker. You cannot know if you are preempted, because if you are, you don't run.

Quoted 2 lines

If we are the worker, we _can_ and _should_ be preempted if something else needs to be scheduled. We don't want to prevent more important tasks to run.

Sure, you want to allow preemption for both the task and the worker. But having the worker check a variable that says if it is itself preempted makes no bloody sense.

Quoted 7 lines

This implies that whatever the task calling sched_pair_cpu set/clear does between set and clear to touch per-cpu data needs to be done with restartable sequences, so the critical section is aborted if the task is preempted. Same goes if the kworker thread is preempted: it preempts the paired task with an IPI, which immediately prevents its user-space to touch the per-cpu data until the kworker is scheduled back and picks the eventually re-queued work for another round.

Sure, if the worker gets preempted we need to preempt the task, and if the task gets preempted, we need to stop the worker.

Quoted 10 lines
Quoted 9 lines

If we're supposed to soak up cycles while @task runs, wth does it allow scheduling in this loop?

Basicaly I think this wants to be something like:

	while (READ_ONCE(task->on_cpu) && READ_ONCE(task->pair_cpu) == cpu)
		cpu_relax();

Nothing more, nothing less.

Quoted 8 lines

The semantic of what you propose is:

- Stop running as soon as @task is not running,
- Stop running as soon as @task is not paired with the cpu serviced by this worker thread.

The semantic I need is:

- Stop running as soon as @task does not need the worker thread,

Can be done by setting pair_cpu to -1

Quoted 1 line

- Stop running as soon as the worker thread is preempted,

This is the bloody worker thread. It cannot know if it is preempted.

Quoted 1 line

- Stop running as soon as @task has exited,

Weirdly enough ->on_cpu is always 0 for dead tasks.

Quoted 1 line

- Stop running as soon as timeout is reached.

I still don't get the timeout, if the pair_cpu is offline, the worker simply shouldn't run.

Quoted 8 lines
Quoted 3 lines

None of this makes sense; task_cpu() will always be >= 0, and if it timed out, it can't be on an offline cpu, because offline CPUs don't run tasks.

There are three scenarios to consider here:

1) The kernel worker thread is handling a CPU which is offline: This explains the need for a "timeout", as discussed earlier.

Still doesn't make sense

Quoted 5 lines

2) The paired task was running on @cpu when task_cpu() is issued, but the CPU is concurrently going offline before we send the IPI: As the comment explains, the preemption of @task is required in order to allow its CPU to go offline makes the IPI redundant.

Tasks' CPU cannot go offline without task beeing preempted. Same for a worker. Either preemption will cycle the state.

Quoted 3 lines

3) The paired task is blocked on @cpu in parallel with the timeout being reached by the worker thread: AFAIU this would cause cpu < 0. Again in this case, the task has been preempted, which means our IPI would be redundant.

task_cpu() is never <0. For blocked tasks, task_cpu() is the last cpu they ran on, which must be a valid cpu. We use it for placement on wakeup.

Quoted 17 lines
Quoted 17 lines

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1851

Quoted 13 lines
kernel/sched/core.c +3 -0
+	preempt_disable();
+	set_current_state(TASK_INTERRUPTIBLE);
+	trace_printk("notify resume block for cpu %d from task %p state 0x%lx\n",

task_pair_cpu,

kernel/sched/core.c +9 -0
+	       current, current->state);
+	WARN_ON_ONCE(current->pair_cpu_worker_active);
+	WRITE_ONCE(current->pair_cpu_need_worker, 1);
+	get_task_struct(current);
+	kthread_init_work(&current->pair_cpu_work, pair_cpu_work_func);
+	kthread_queue_work(cpum->worker, &current->pair_cpu_work);
+	current->pair_cpu_queued_work = 1;
+	preempt_enable();
+	schedule();

Did you consider spurious wakeups? A naked schedule() pretty much always wrong. They need to come in loops and testing variables.

Quoted 7 lines
Quoted 4 lines

Basically this should be something like:

	rcu_wait_event(.., smp_load_acquire(&current->pair_cpu_worker_active),
			TASK_INTERRUPTIBLE);

This schedule() is already in a loop: __sched_pair_cpu_handle_notify_resume() is called from exit_to_usermode_loop().

urgh..

Quoted 5 lines
Quoted 1 line

Also, I can't help but think this should've been a task_work.

"queuing" that work is needed from IPI context and from finish task switch. As long as we can queue task_work from atomic context, we could indeed use that instead. The struct callback_head would sit in the task struct.

It's a cmpxchg add + set_notify_resume() if the list was empty.

Quoted 2 lines

Doing that change means that I cannot rely on the return to usermode loop to call this again after schedule, so I should put that in a loop myself.

Yes, using a wait-event style API is much easier to understand.

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
kernel/sched/core.c +13 -0
+/*
+ * If we preempt the cpu mutex worker, we need to IPI the CPU
+ * running the thread currently associated to it before scheduling
+ * other tasks.
+ *
+ * This only targets pair_cpu worker for online cpus.
+ */
+static void pair_cpu_finish_switch_worker(struct task_struct *prev)
+{
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, smp_processor_id());
+	struct task_struct *running_task;
+	int cpu;
+
Quoted 8 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3366

Quoted 5 lines
kernel/sched/core.c +5 -0
+	if (!cpum->worker || prev != cpum->worker->task)
+		return;
+	running_task = READ_ONCE(cpum->running);
+	if (!running_task)
+		return;

You're doing this for _every_ context switch, the above is unconditional new lines touched and a big no-no.

Quoted 6 lines

The only state I need to check here is whether the scheduler is preempting a thread which is a pair-cpu kworker thread. This could be a simple flag within an already cache-hot cacheline in the task struct. The rest then becomes a slow-path.

Would that solve your performance concern ?

The other alternative is using a preempt_notifier for the worker I suppose.

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
kernel/sched/core.c +5 -0
+/*
+ * If we preempt a task currently associated with a cpu mutex worker,
+ * we need to tell the worker to stop using cpu time.
+ */
+static void pair_cpu_finish_switch_task(struct task_struct *prev, long

prev_state)

(quoted diff) +2 -0
+{
+	int prev_pair_cpu;
Quoted 8 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3411

Quoted 5 lines
kernel/sched/core.c +5 -0
+
+	prev_pair_cpu = READ_ONCE(prev->pair_cpu);
+	if (prev_pair_cpu < 0 || !READ_ONCE(prev->pair_cpu_need_worker)
+	    || !READ_ONCE(prev->pair_cpu_worker_active))
+		return;

Again, you're doing this every context switch, touching memory we didn't used to touch.

Quoted 2 lines

Same argument about placing this in a single "hot path" flag in the task struct. Would that address your performance concerns ?

preempt_notifier could work here too I suppose, install it on yourself when you do the pear syscall and take it away again when you're finished with it.

Quoted 10 lines
Quoted 8 lines

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4386

Quoted 3 lines
kernel/sched/core.c +0 -0
 	if (likely(prev != next)) {
 		rq->nr_switches++;

I think if you use a task instead of a kthread_work things become much simpler. Regardless, you have too much state which mightily confuses things.

Do you mean "use a task_work" or really "use a task" ?

I meant replace the kthread_work with a full kthread, a task.

Quoted 3 lines

I used a kthread_work to eliminate the need to re-implement the work handling logic. What would be the benefit of using a task rather than kthread_work here ?

If the worker is a dedicated task, it is much easier to see if it is running, simply look at worker->on_cpu. Similarly, if it is a full task the fact of ->on_cpu being 0 means it has scheduled, which implies barriers.

But I think I see why, you can have multiple tasks requiring the same worker. I'd overlooked that this morning.

Quoted 5 lines
Quoted 3 lines

As with all the other approaches, I don't love this.. All of this complexity is because sodding hotplug -- something that should really never happen.

Actually, it's not only hotplug. It's hotplug _and_ cgroup cpusets.

Argh.. I'd forgotten cpusets again :-(

Quoted 8 lines
Quoted 3 lines

- make fcntl(F_*SIG) work on /sys/devices/system/cpu/cpuN/online such that we receive a signal when the state changes, and before userspace starts. The signal triggers rseq restart.

Just knowing a CPU is offline is not enough. We then need to be able to synchronize concurrent updates to that per-CPU data structure while the CPU is offline, in a way which is also safe against that CPU coming back online again.

That's exactly what that signal would do. It would send SIGIO when the state changes.

So you want to access CPU-n's data, you open that file, register a signal and read it's state, if offline, you good, do the rseq. If it suddenly decides to come back online, you're guaranteed that SIGIO before it reaches userspace.

The nice thing is that it's all R/O so available to normal users, you don't have to write to the file.

Quoted 3 lines

Also it does not solve the issues associated with cgroup cpusets, where a CPU may very well be online, but not part of our thread's allowed cpu mask.

Let's just say I violated the CoC...

Quoted 9 lines

We do not want to override the affinity restricted by cgroups because we don't want to hurt performance characteristics of another partition of the system.

The sched_pair_cpu approach has the benefit of allowing us to touch per-cpu data of a given CPU without requiring to run on that CPU, which ensures that we do not thrash the cpu cache of cpus on which a thread is not allowed to run. It takes care of issues caused by both cgroup cpusets and cpu hotplug.

But now I worry that your thing allows escaping the cgroup contraints, you can perturb random CPUs you're not allowed on. That's a really bad 'feature'.

Offline cpus are okay, because you don't actually need to do anything as long as they're offline, but restricted CPUs we really should not be touching, not even a little.

#7 lore raw
Quoted 2 lines · ----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

On Wed, Jun 24, 2020 at 02:31:33PM -0400, Mathieu Desnoyers wrote:

[...]

Quoted 26 lines
Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
kernel/sched/core.c +4 -0
+	/*
+	 * Consume CPU time as long as an associated task is running on another
+	 * CPU.
+	 */

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1771

Quoted 10 lines
kernel/sched/core.c +10 -0
+	while (READ_ONCE(task->pair_cpu_need_worker)
+	       && !READ_ONCE(cpum->worker_preempted)
+	       && task->state != TASK_DEAD) {
+		timeout = ktime_sub(ktime_get(), time_begin) > MAX_SCHED_PAIR_CPU_WORK_NS;
+		if (timeout)
+			break;
+		touch_softlockup_watchdog();
+		cond_resched();
+		cpu_relax();
+	}

Why does it have a timeout?

Quoted 7 lines

The timeout is used to handle case where the kernel worker thread is handling the per-cpu data of an offline CPU. In that peculiar case, the worker thread will be running on a CPU which is not the offline CPU, so no preemption will set the "worker_preempted" flag in the struct pair_cpu as long as the CPU is offline. The timeout ensures that a single worker task does not stay arbitrarily long as "current", and gets kicked away to leave room for other tasks which may also need to access that offlined cpu's per-cpu data.

Quoted 2 lines

But if the CPU is offline, you don't actually need to do anything. So that seems unfortunate.

Quite the opposite: if the CPU is offline, we need to serialize all tasks which want to touch its per-cpu data concurrently. So in that situation, the kworker thread acts as a request queuing mechanism, allowing only a single task to touch that per-cpu data at a given time.

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

And this is a radically weird approach to
testing if @task is currently running. And if we're the worker, we can't be preempted, per definition.

Note that the "worker_preempted" test is really testing whether the _worker_ has been preempted.

But this is the worker. You cannot know if you are preempted, because if you are, you don't run.

The worker can check if it _has been_ preempted since it began handling the current work item. Setting the "worker_preempted" flag is done by the finish task switch code.

Quoted 7 lines
Quoted 3 lines

If we are the worker, we _can_ and _should_ be preempted if something else needs to be scheduled. We don't want to prevent more important tasks to run.

Sure, you want to allow preemption for both the task and the worker. But having the worker check a variable that says if it is itself preempted makes no bloody sense.

Fair point.

Quoted 11 lines
Quoted 8 lines

This implies that whatever the task calling sched_pair_cpu set/clear does between set and clear to touch per-cpu data needs to be done with restartable sequences, so the critical section is aborted if the task is preempted. Same goes if the kworker thread is preempted: it preempts the paired task with an IPI, which immediately prevents its user-space to touch the per-cpu data until the kworker is scheduled back and picks the eventually re-queued work for another round.

Sure, if the worker gets preempted we need to preempt the task, and if the task gets preempted, we need to stop the worker.

I'm starting to grasp where you are getting: this "worker_preempted" flag is redundant with the fact that the finish task switch code IPI the paired task when it preempts the worker thread. So by side-effect, that paired task will set its "need worker" to 0, which will allow the worker thread to exit the loop.

Indeed, I think I can just remove that worker_preempted flag.

Quoted 17 lines
Quoted 17 lines
Quoted 10 lines

If we're supposed to soak up cycles while @task runs, wth does it allow scheduling in this loop?

Basicaly I think this wants to be something like:

	while (READ_ONCE(task->on_cpu) && READ_ONCE(task->pair_cpu) == cpu)
		cpu_relax();

Nothing more, nothing less.

The semantic of what you propose is:

- Stop running as soon as @task is not running,
- Stop running as soon as @task is not paired with the cpu serviced by this worker thread.

Quoted 5 lines
Quoted 3 lines

The semantic I need is:

- Stop running as soon as @task does not need the worker thread,

Can be done by setting pair_cpu to -1

The state "pair_cpu >= 0" must last from sched_pair_cpu(SET) to sched_pair_cpu(CLEAR).

The state "need worker" needs to be cleared when the paired task is preempted. It lets the worker thread know it must break its loop and stop handling this work, so the paired task can enqueue further work for a later attempt. This provides some fairness across multiple tasks using sched_pair_cpu on the same cpu number. Otherwise a task could keep the pairing set for a very long time and prevent progress from other tasks paired to the same cpu.

So I'm afraid the two states cannot be easily combined.

Quoted 4 lines
Quoted 2 lines

- Stop running as soon as the worker thread is preempted,

This is the bloody worker thread. It cannot know if it is preempted.

As discussed above, it can know if it _has been_ preempted since beginning of work item handling, but this happens to be redundant with checking the "need worker" flag, so I'll remove the "worker_preempted" flag.

Quoted 3 lines
Quoted 1 line

- Stop running as soon as @task has exited,

Weirdly enough ->on_cpu is always 0 for dead tasks.

I don't think we can use "->on_cpu == 0" to break the worker thread's loop.

The paired task, before returning to user-space, will enqueue the kwork before blocking. Then, the kworker thread will eventually awaken the task. However, AFAIU nothing guarantees that the task reaches a ->on_cpu==1 state before the kworker thread starts its loop. It may take some time for the scheduler to really schedule that task if there are more important things to run on that runqueue.

Quoted 5 lines
Quoted 2 lines

- Stop running as soon as timeout is reached.

I still don't get the timeout, if the pair_cpu is offline, the worker simply shouldn't run.

If we don't keep a kworker for offlined cpus, then we need _something else_ to act as a queue manager, ensuring that only a single queued work is allowed at any given time. Because we can have many tasks which try to touch the offline CPU's per-cpu data concurrently, and we need to serialize that as if everyone is running on the same runqueue.

The simple trick I use here is to keep using the same kworker thread which is tied to the offline cpu, and keep it handling work for the offline CPU. I let it run wherever it wants.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

None of this makes sense; task_cpu() will always be >= 0, and if it timed out, it can't be on an offline cpu, because offline CPUs don't run tasks.

Quoted 7 lines
Quoted 5 lines

There are three scenarios to consider here:

1) The kernel worker thread is handling a CPU which is offline: This explains the
need for a "timeout", as discussed earlier.

Still doesn't make sense

(replied above)

Quoted 10 lines
Quoted 7 lines

2) The paired task was running on @cpu when task_cpu() is issued, but the CPU is concurrently going offline before we send the IPI: As the comment explains, the preemption of @task is required in order to allow its CPU to go offline makes the IPI
redundant.

Tasks' CPU cannot go offline without task beeing preempted. Same for a worker. Either preemption will cycle the state.

Exactly. This is why I comment that a failed IPI is not an issue, because it is redundant.

Quoted 9 lines
Quoted 5 lines

3) The paired task is blocked on @cpu in parallel with the timeout being reached by
the worker thread: AFAIU this would cause cpu < 0. Again in this case, the task has been preempted, which means our IPI would be redundant.

task_cpu() is never <0. For blocked tasks, task_cpu() is the last cpu they ran on, which must be a valid cpu. We use it for placement on wakeup.

I did not notice this, good point ! I'll fix that.

Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines
kernel/sched/core.c +3 -0
+	preempt_disable();
+	set_current_state(TASK_INTERRUPTIBLE);
+	trace_printk("notify resume block for cpu %d from task %p state 0x%lx\n",

task_pair_cpu,

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1851

Quoted 9 lines
kernel/sched/core.c +9 -0
+	       current, current->state);
+	WARN_ON_ONCE(current->pair_cpu_worker_active);
+	WRITE_ONCE(current->pair_cpu_need_worker, 1);
+	get_task_struct(current);
+	kthread_init_work(&current->pair_cpu_work, pair_cpu_work_func);
+	kthread_queue_work(cpum->worker, &current->pair_cpu_work);
+	current->pair_cpu_queued_work = 1;
+	preempt_enable();
+	schedule();

Did you consider spurious wakeups? A naked schedule() pretty much always wrong. They need to come in loops and testing variables.

Quoted 7 lines
Quoted 4 lines

Basically this should be something like:

	rcu_wait_event(.., smp_load_acquire(&current->pair_cpu_worker_active),
			TASK_INTERRUPTIBLE);

This schedule() is already in a loop: __sched_pair_cpu_handle_notify_resume() is called from exit_to_usermode_loop().

urgh..

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Also, I can't help but think this should've been a task_work.

"queuing" that work is needed from IPI context and from finish task switch. As long as we can queue task_work from atomic context, we could indeed use that instead. The struct callback_head would sit in the task struct.

It's a cmpxchg add + set_notify_resume() if the list was empty.

That works.

Quoted 5 lines
Quoted 3 lines

Doing that change means that I cannot rely on the return to usermode loop to call this again after schedule, so I should put that in a loop myself.

Yes, using a wait-event style API is much easier to understand.

Changing this custom wait/wakeup to a wait-event API, one question arises:
Should we use a TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state ? I especially wonder how we should handle interruption by signal within this task_work, given that this can be running on top of return to userspace on top of pretty much anything, including an interrupt. So returning things like ERESTARTSYS or EINTR does not seem possible here.

Quoted 28 lines
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
kernel/sched/core.c +13 -0
+/*
+ * If we preempt the cpu mutex worker, we need to IPI the CPU
+ * running the thread currently associated to it before scheduling
+ * other tasks.
+ *
+ * This only targets pair_cpu worker for online cpus.
+ */
+static void pair_cpu_finish_switch_worker(struct task_struct *prev)
+{
+	struct pair_cpu *cpum = per_cpu_ptr(&pair_cpu, smp_processor_id());
+	struct task_struct *running_task;
+	int cpu;
+
Quoted 8 lines

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3366

Quoted 5 lines
kernel/sched/core.c +5 -0
+	if (!cpum->worker || prev != cpum->worker->task)
+		return;
+	running_task = READ_ONCE(cpum->running);
+	if (!running_task)
+		return;

You're doing this for _every_ context switch, the above is unconditional new lines touched and a big no-no.

The only state I need to check here is whether the scheduler is preempting a thread which is a pair-cpu kworker thread. This could be a simple flag within an already cache-hot cacheline in the task struct. The rest then becomes a slow-path.

Quoted 4 lines
Quoted 1 line

Would that solve your performance concern ?

The other alternative is using a preempt_notifier for the worker I suppose.

Good point!

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
kernel/sched/core.c +5 -0
+/*
+ * If we preempt a task currently associated with a cpu mutex worker,
+ * we need to tell the worker to stop using cpu time.
+ */
+static void pair_cpu_finish_switch_task(struct task_struct *prev, long

prev_state)

kernel/sched/core.c static inline void finish_lock_switch(struct rq *rq) · line 3411

Quoted 7 lines
kernel/sched/core.c +7 -0
+{
+	int prev_pair_cpu;
+
+	prev_pair_cpu = READ_ONCE(prev->pair_cpu);
+	if (prev_pair_cpu < 0 || !READ_ONCE(prev->pair_cpu_need_worker)
+	    || !READ_ONCE(prev->pair_cpu_worker_active))
+		return;

Again, you're doing this every context switch, touching memory we didn't used to touch.

Quoted 6 lines
Quoted 2 lines

Same argument about placing this in a single "hot path" flag in the task struct. Would that address your performance concerns ?

preempt_notifier could work here too I suppose, install it on yourself when you do the pear syscall and take it away again when you're finished with it.

Good point, will do!

Quoted 18 lines
Quoted 11 lines
Quoted 9 lines

kernel/sched/core.c static void __sched notrace __schedule(bool preempt) · line 4386

Quoted 4 lines
kernel/sched/core.c +0 -0
 	if (likely(prev != next)) {
 		rq->nr_switches++;

I think if you use a task instead of a kthread_work things become much simpler. Regardless, you have too much state which mightily confuses things.

Do you mean "use a task_work" or really "use a task" ?

I meant replace the kthread_work with a full kthread, a task.

Quoted 3 lines

I used a kthread_work to eliminate the need to re-implement the work handling logic. What would be the benefit of using a task rather than kthread_work here ?

Quoted 7 lines

If the worker is a dedicated task, it is much easier to see if it is running, simply look at worker->on_cpu. Similarly, if it is a full task the fact of ->on_cpu being 0 means it has scheduled, which implies barriers.

But I think I see why, you can have multiple tasks requiring the same worker. I'd overlooked that this morning.

OK, so I'll stick with a kworker for now.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

As with all the other approaches, I don't love this.. All of this complexity is because sodding hotplug -- something that should really never happen.

Actually, it's not only hotplug. It's hotplug _and_ cgroup cpusets.

Argh.. I'd forgotten cpusets again :-(

Yes, cpusets make things "interesting". ;)

Quoted 13 lines
Quoted 9 lines
Quoted 4 lines

- make fcntl(F_*SIG) work on /sys/devices/system/cpu/cpuN/online such that we receive a signal when the state changes, and before userspace starts. The signal triggers rseq restart.

Just knowing a CPU is offline is not enough. We then need to be able to synchronize concurrent updates to that per-CPU data structure while the CPU is offline, in a way which is also safe against that CPU coming back online again.

That's exactly what that signal would do. It would send SIGIO when the state changes.

Quoted 7 lines

So you want to access CPU-n's data, you open that file, register a signal and read it's state, if offline, you good, do the rseq. If it suddenly decides to come back online, you're guaranteed that SIGIO before it reaches userspace.

The nice thing is that it's all R/O so available to normal users, you don't have to write to the file.

So let's say you have two threads trying to access (offline) CPU-n's data with that algorithm concurrently. How are they serialized with each other ?

Quoted 6 lines
Quoted 4 lines

Also it does not solve the issues associated with cgroup cpusets, where a CPU may very well be online, but not part of our thread's allowed cpu mask.

Let's just say I violated the CoC...

:-)

Quoted 11 lines
Quoted 10 lines

We do not want to override the affinity restricted by cgroups because we don't want to hurt performance characteristics of another partition of the system.

The sched_pair_cpu approach has the benefit of allowing us to touch per-cpu data of a given CPU without requiring to run on that CPU, which ensures that we do not thrash the cpu cache of cpus on which a thread is not allowed to run. It takes care of issues caused by both cgroup cpusets and cpu hotplug.

Quoted 7 lines

But now I worry that your thing allows escaping the cgroup contraints, you can perturb random CPUs you're not allowed on. That's a really bad 'feature'.

Offline cpus are okay, because you don't actually need to do anything as long as they're offline, but restricted CPUs we really should not be touching, not even a little.

With sched_pair_cpu, the paired task never needs to run on the target CPU. The kworker thread runs on the target CPU in the same way other existing worker threads run today, e.g. the ones handling RCU callbacks. AFAIK the priority of those threads can be configured by a system administrator.

Are there additional steps we should take to minimize the impact of this worker thread ? In the same way "no rcu callbacks" CPU can be configured at boot time, we could have "no sched pair cpu" configured at boot, which would prevent sched_pair_cpu system calls from targeting that CPU entirely, and not spawn any kworker on that cpu.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#8 lore raw
Quoted 8 lines · ----- On Jun 25, 2020, at 10:56 AM, Mathieu Desnoyers…
Quoted 2 lines · ----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

On Wed, Jun 24, 2020 at 02:31:33PM -0400, Mathieu Desnoyers wrote:

[...]

Quoted 2 lines

The other alternative is using a preempt_notifier for the worker I suppose.

[...]

Quoted 4 lines
Quoted 4 lines

preempt_notifier could work here too I suppose, install it on yourself when you do the pear syscall and take it away again when you're finished with it.

The issue I currently have with preempt notifiers is that I need to send an IPI from a sched_out notifier, which has interrupts off and hold the rq lock. smp_call_function_single() warns due to irq off, and indeed it triggers deadlocks.

Before using preempt notifiers, I was touching the "prev" task after irqs were reenabled and rq lock was released, which allowed me to send an IPI from that context.

Any thoughts on how to best solve this ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#9 lore raw
Quoted 22 lines · ----- On Jun 25, 2020, at 12:34 PM, Mathieu Desnoyers…

----- On Jun 25, 2020, at 10:56 AM, Mathieu Desnoyers

Quoted 8 lines · [email protected] wrote:
Quoted 2 lines · ----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

On Wed, Jun 24, 2020 at 02:31:33PM -0400, Mathieu Desnoyers wrote:

[...]

Quoted 2 lines

The other alternative is using a preempt_notifier for the worker I suppose.

[...]

Quoted 4 lines
Quoted 4 lines

preempt_notifier could work here too I suppose, install it on yourself when you do the pear syscall and take it away again when you're finished with it.

The issue I currently have with preempt notifiers is that I need to send an IPI from a sched_out notifier, which has interrupts off and hold the rq lock. smp_call_function_single() warns due to irq off, and indeed it triggers deadlocks.

Quoted 5 lines

Before using preempt notifiers, I was touching the "prev" task after irqs were reenabled and rq lock was released, which allowed me to send an IPI from that context.

Any thoughts on how to best solve this ?

I think I may have found a way out of this: I may not need to use smp_call_function_single() at all.

When preempting a paired task, I think we can rely on memory barrier at the beginning of scheduling of the paired task to match the memory barrier at the end of scheduling of the kworker thread to provide memory ordering. Therefore, the IPI is not needed at all in this case.

When preempting the kworker thread, things are a bit trickier. AFAIU I can simply queue task work on the paired task directly without an IPI, and then use kick_process() on the paired task.

The remaining concern is whether kick_process() (and thus smp_send_reschedule()) is sufficient to guarantee a memory barrier before smp_send_reschedule returns ? I suspect not, because it only raises the IPI, and does not appear to wait for its handler to complete. In that case I need a release on the paired task and an acquire in sched_out of the kworker. The memory barrier at the end of schedule fulfills the acquire, but I don't see how the acquire is done on the paired task, because execution of its scheduler does not necessarily happen immediately when the IPI is raised.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#10 lore raw
Quoted 45 lines · ----- On Jun 26, 2020, at 11:16 AM, Mathieu Desnoyers…

----- On Jun 25, 2020, at 12:34 PM, Mathieu Desnoyers

Quoted 22 lines · [email protected] wrote:

----- On Jun 25, 2020, at 10:56 AM, Mathieu Desnoyers

Quoted 8 lines · [email protected] wrote:
Quoted 2 lines · ----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

On Wed, Jun 24, 2020 at 02:31:33PM -0400, Mathieu Desnoyers wrote:

[...]

Quoted 2 lines

The other alternative is using a preempt_notifier for the worker I suppose.

[...]

Quoted 4 lines
Quoted 4 lines

preempt_notifier could work here too I suppose, install it on yourself when you do the pear syscall and take it away again when you're finished with it.

The issue I currently have with preempt notifiers is that I need to send an IPI from a sched_out notifier, which has interrupts off and hold the rq lock. smp_call_function_single() warns due to irq off, and indeed it triggers deadlocks.

Quoted 5 lines

Before using preempt notifiers, I was touching the "prev" task after irqs were reenabled and rq lock was released, which allowed me to send an IPI from that context.

Any thoughts on how to best solve this ?

I think I may have found a way out of this: I may not need to use smp_call_function_single() at all.

When preempting a paired task, I think we can rely on memory barrier at the beginning of scheduling of the paired task to match the memory barrier at the end of scheduling of the kworker thread to provide memory ordering. Therefore,
the IPI is not needed at all in this case.

When preempting the kworker thread, things are a bit trickier. AFAIU I can simply
queue task work on the paired task directly without an IPI, and then use kick_process() on the paired task.

Quoted 10 lines

The remaining concern is whether kick_process() (and thus smp_send_reschedule()) is sufficient to guarantee a memory barrier before smp_send_reschedule returns ? I suspect not, because it only raises the IPI, and does not appear to wait for its handler to complete. In that case I need a release on the paired task and an acquire in sched_out of the kworker. The memory barrier at the end of schedule
fulfills the acquire, but I don't see how the acquire is done on the paired task,
because execution of its scheduler does not necessarily happen immediately when the IPI is raised.

Hrm, smp_cond_load_acquire(&running_task->on_cpu, !VAL); is probably all I need after that kick_process.

Thanks,

Mathieu

Quoted 9 lines

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#11 lore raw
In reply to Mathieu Desnoyers
Quoted 1 line · Mathieu Desnoyers

----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

I'll try and read the earlier bit later, I can't think today.

Quoted 8 lines
Quoted 8 lines

That's exactly what that signal would do. It would send SIGIO when the state changes.

So you want to access CPU-n's data, you open that file, register a signal and read it's state, if offline, you good, do the rseq. If it suddenly decides to come back online, you're guaranteed that SIGIO before it reaches userspace.

Quoted 5 lines
Quoted 2 lines

The nice thing is that it's all R/O so available to normal users, you don't have to write to the file.

So let's say you have two threads trying to access (offline) CPU-n's data with that algorithm concurrently. How are they serialized with each other ?

Also implement F_SETLK or something :-)

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines

We do not want to override the affinity restricted by cgroups because we don't want to hurt performance characteristics of another partition of the system.

The sched_pair_cpu approach has the benefit of allowing us to touch per-cpu data of a given CPU without requiring to run on that CPU, which ensures that we do not thrash the cpu cache of cpus on which a thread is not allowed to run. It takes care of issues caused by both cgroup cpusets and cpu hotplug.

But now I worry that your thing allows escaping the cgroup contraints, you can perturb random CPUs you're not allowed on. That's a really bad 'feature'.

Quoted 8 lines
Quoted 3 lines

Offline cpus are okay, because you don't actually need to do anything as long as they're offline, but restricted CPUs we really should not be touching, not even a little.

With sched_pair_cpu, the paired task never needs to run on the target CPU. The kworker thread runs on the target CPU in the same way other existing worker threads run today, e.g. the ones handling RCU callbacks. AFAIK the priority of those threads can be configured by a system administrator.

Ah, but the critical difference is that all those are only ever ran if the initial work was initialized on _that_ CPU to begin with. Consider an isolated CPU that's spinning in userspace, it would _never_ get any kthreads running.

Except now you can, and you even want this system call to be unpriv.

It utterly and completely wrecks NOHZ_FULL.

Quoted 5 lines

Are there additional steps we should take to minimize the impact of this worker thread ? In the same way "no rcu callbacks" CPU can be configured at boot time, we could have "no sched pair cpu" configured at boot, which would prevent sched_pair_cpu system calls from targeting that CPU entirely, and not spawn any kworker on that cpu.

No, no, no! "at boot time" is an utter trainwreck. I've been trying to get NOHZ_FULL runtime configurable. This means that your cpuset can change at runtime and the CPU you tought you had now is a NOHZ_FULL CPU.

We must not allow pears on it.

I'm thinking that the best option might be to treat CPUs outside of your cpuset the same as offline CPUs. That more-or-less requires that tasks outside of your cpuset partition don't have access to your shared memory, but that isn't an entirely insane assumption.

If you want to share memory across cpuset partitions, you get to keep the pieces.

And the nice thing about offline, is that you don't actually need to run anything. You only need some exclusion thing (and using a spin-loop on a random other CPU for that is bloody insane).

#12 lore raw
Quoted 13 lines · ----- On Jun 26, 2020, at 12:00 PM, Peter Zijlstra [email protected] wrote:
Quoted 1 line · Mathieu Desnoyers

----- On Jun 24, 2020, at 3:50 PM, Peter Zijlstra [email protected] wrote:

I'll try and read the earlier bit later, I can't think today.

Quoted 8 lines
Quoted 8 lines

That's exactly what that signal would do. It would send SIGIO when the state changes.

So you want to access CPU-n's data, you open that file, register a signal and read it's state, if offline, you good, do the rseq. If it suddenly decides to come back online, you're guaranteed that SIGIO before it reaches userspace.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The nice thing is that it's all R/O so available to normal users, you don't have to write to the file.

So let's say you have two threads trying to access (offline) CPU-n's data with that algorithm concurrently. How are they serialized with each other ?

Also implement F_SETLK or something :-)

I don't see this being available to non-root users.

Quoted 39 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines

We do not want to override the affinity restricted by cgroups because we don't want to hurt performance characteristics of another partition of the system.

The sched_pair_cpu approach has the benefit of allowing us to touch per-cpu data of a given CPU without requiring to run on that CPU, which ensures that we do not thrash the cpu cache of cpus on which a thread is not allowed to run. It takes care of issues caused by both cgroup cpusets and cpu hotplug.

But now I worry that your thing allows escaping the cgroup contraints, you can perturb random CPUs you're not allowed on. That's a really bad 'feature'.

Quoted 8 lines
Quoted 3 lines

Offline cpus are okay, because you don't actually need to do anything as long as they're offline, but restricted CPUs we really should not be touching, not even a little.

With sched_pair_cpu, the paired task never needs to run on the target CPU. The kworker thread runs on the target CPU in the same way other existing worker threads run today, e.g. the ones handling RCU callbacks. AFAIK the priority of those threads can be configured by a system administrator.

Ah, but the critical difference is that all those are only ever ran if the initial work was initialized on _that_ CPU to begin with. Consider an isolated CPU that's spinning in userspace, it would _never_ get any kthreads running.

Except now you can, and you even want this system call to be unpriv.

It utterly and completely wrecks NOHZ_FULL.

Quoted 5 lines

Are there additional steps we should take to minimize the impact of this worker thread ? In the same way "no rcu callbacks" CPU can be configured at boot time, we could have "no sched pair cpu" configured at boot, which would prevent sched_pair_cpu system calls from targeting that CPU entirely, and not spawn any kworker on that cpu.

Quoted 5 lines

No, no, no! "at boot time" is an utter trainwreck. I've been trying to get NOHZ_FULL runtime configurable. This means that your cpuset can change at runtime and the CPU you tought you had now is a NOHZ_FULL CPU.

We must not allow pears on it.

One possibility is to simply treat NOHZ_FULL cpus as offline from the perspective of sched_pair_cpu: no kworker thread on those, and the queuing is handled elsewhere. As long as rseq is not be used on NOHZ_FULL cpus concurrently with sched_pair_cpu targeting those CPUs, it would work.

Quoted 7 lines

I'm thinking that the best option might be to treat CPUs outside of your cpuset the same as offline CPUs. That more-or-less requires that tasks outside of your cpuset partition don't have access to your shared memory, but that isn't an entirely insane assumption.

If you want to share memory across cpuset partitions, you get to keep the pieces.

The main issue is that cpuset partitions are changed dynamically at runtime by external "manager" processes (e.g. Android). To make things even more interesting, cpusets support both "process" and "threaded" domain types.

There are quite a few scenarios which worry me with this approach, e.g.:

A) rseq and sched_pair_cpu are used by a memory allocator within a process, and specific threads of that process eventually have their cpuset changed to exclude some cpus (with cpusets applied per-thread rather than per-process). Now the memory allocator needs to touch the per-cpu data of specific cpus to which it does not have access from a given thread while other threads within the process still use it concurrently.

B) A tracer ring buffer works on per-cpu data in shared memory across processes with rseq and sched_pair_cpu, and races on per-cpu data because an external manager process concurrently applies different cpusets (with process domain) to processes interacting over that shared memory.

Quoted 3 lines

And the nice thing about offline, is that you don't actually need to run anything. You only need some exclusion thing (and using a spin-loop on a random other CPU for that is bloody insane).

Indeed, for the offline case, the kworker really does not need to keep burning CPU time. I should eventually have it sleep for a while instead, or have the work queues for all offline cpus handled by a single kworker for the entire system.

So treating NOHZ_FULL cpus as offline, I'm all good with that. Treating cpus which are not in the cpuset as offline, not so much.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com