Skip to contentSkip to conversation

sched: Try and address some recent-ish regressions

RFC PATCH In review Peter Zijlstra 9 participants 33 messages May 20, 2025 – Jun 16, 2025 Scheduler

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover sched: Try and address some recent-ish regressions

Hi!

So Chris poked me about how they're having a wee performance drop after around 6.11. He's extended his schbench tool to mimic the workload in question.

Specifically the commandline given:

schbench -L -m 4 -M auto -t 128 -n 0 -r 60

This benchmark wants to stay on a single (large) LLC (Chris, perhaps add an option to start the CPU mask with /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list or something). Both the machine Chris has (SKL, 20+ cores per LLC) and the machines I ran this on (SKL,SPR 20+ cores) are Intel, AMD has smaller LLC and the problem wasn't as pronounced there.

Use performance CPU governor (as always when benchmarking). Also, if the test results are unstable as all heck, disable turbo.

After a fair amount of tinkering I managed to reproduce on my SPR and Thomas' SKL. The SKL would only give usable numbers with the second socket offline and turbo disabled -- YMMV.

Chris further provided a bisect into the DELAY_DEQUEUE patches and a bisect leading to commit 5f6bd380c7bd ("sched/rt: Remove default bandwidth control") -- which enables the dl_server by default.

SKL (performance, no_turbo):

schbench-6.9.0-1.txt:average rps: 2040360.55
schbench-6.9.0-2.txt:average rps: 2038846.78
schbench-6.9.0-3.txt:average rps: 2037892.28

schbench-6.15.0-rc6+-1.txt:average rps: 1907718.18 schbench-6.15.0-rc6+-2.txt:average rps: 1906931.07 schbench-6.15.0-rc6+-3.txt:average rps: 1903190.38

schbench-6.15.0-rc6+-dirty-1.txt:average rps: 2002224.78 schbench-6.15.0-rc6+-dirty-2.txt:average rps: 2007116.80 schbench-6.15.0-rc6+-dirty-3.txt:average rps: 2005294.57

schbench-6.15.0-rc6+-dirty-delayed-1.txt:average rps: 2011282.15 schbench-6.15.0-rc6+-dirty-delayed-2.txt:average rps: 2016347.10 schbench-6.15.0-rc6+-dirty-delayed-3.txt:average rps: 2014515.47

schbench-6.15.0-rc6+-dirty-delayed-default-1.txt:average rps: 2042169.00 schbench-6.15.0-rc6+-dirty-delayed-default-2.txt:average rps: 2032789.77 schbench-6.15.0-rc6+-dirty-delayed-default-3.txt:average rps: 2040313.95

SPR (performance):

schbench-6.9.0-1.txt:average rps: 2975450.75
schbench-6.9.0-2.txt:average rps: 2975464.38
schbench-6.9.0-3.txt:average rps: 2974881.02

schbench-6.15.0-rc6+-1.txt:average rps: 2882537.37 schbench-6.15.0-rc6+-2.txt:average rps: 2881658.70 schbench-6.15.0-rc6+-3.txt:average rps: 2884293.37

schbench-6.15.0-rc6+-dl_server-1.txt:average rps: 2924423.18 schbench-6.15.0-rc6+-dl_server-2.txt:average rps: 2920422.63

schbench-6.15.0-rc6+-dirty-1.txt:average rps: 3011540.97 schbench-6.15.0-rc6+-dirty-2.txt:average rps: 3010124.10

schbench-6.15.0-rc6+-dirty-delayed-1.txt:average rps: 3030883.15 schbench-6.15.0-rc6+-dirty-delayed-2.txt:average rps: 3031627.05

schbench-6.15.0-rc6+-dirty-delayed-default-1.txt:average rps: 3053005.98 schbench-6.15.0-rc6+-dirty-delayed-default-2.txt:average rps: 3052972.80

As can be seen, the SPR is much easier to please than the SKL for whatever reason. I'm thinking we can make TTWU_QUEUE_DELAYED default on, but I suspect TTWU_QUEUE_DEFAULT might be a harder sell -- we'd need to run more than this one benchmark.

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

author maintainer
#1 lore raw

RFC PATCH 1/5 sched/deadline: Less agressive dl_server handling

Chris reported that commit 5f6bd380c7bd ("sched/rt: Remove default bandwidth control") caused a significant dip in his favourite benchmark of the day. Simply disabling dl_server cured things.

His workload hammers the 0->1, 1->0 transitions, and the dl_server_{start,stop}() overhead kills it -- fairly obviously a bad idea in hind sight and all that.

Change things around to only disable the dl_server when there has not been a fair task around for a whole period. Since the default period is 1 second, this ensures the benchmark never trips this, overhead gone.

Fixes 557a6bfc662c ("sched/fair: Add trivial fair server")
---
 include/linux/sched.h   |    1 +
 kernel/sched/deadline.c |   31 +++++++++++++++++++++++++++----
 2 files changed, 28 insertions(+), 4 deletions(-)
include/linux/sched.h +1 -0
@@ -702,6 +702,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer	  : 1;
 	unsigned int			dl_defer_armed	  : 1;
 	unsigned int			dl_defer_running  : 1;
+	unsigned int			dl_server_idle    : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
kernel/sched/deadline.c +27 -4
@@ -1215,6 +1215,8 @@ static void __push_dl_task(struct rq *rq
 /* a defer timer will not be reset if the runtime consumed was < dl_server_min_res */
 static const u64 dl_server_min_res = 1 * NSEC_PER_MSEC;
 
+static bool dl_server_stopped(struct sched_dl_entity *dl_se);
+
 static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = rq_of_dl_se(dl_se);
@@ -1234,6 +1236,7 @@ static enum hrtimer_restart dl_server_ti
 
 		if (!dl_se->server_has_tasks(dl_se)) {
 			replenish_dl_entity(dl_se);
+			dl_server_stopped(dl_se);
 			return HRTIMER_NORESTART;
 		}
 
@@ -1639,8 +1642,10 @@ void dl_server_update_idle_time(struct r
 void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
 {
 	/* 0 runtime = fair server disabled */
-	if (dl_se->dl_runtime)
+	if (dl_se->dl_runtime) {
+		dl_se->dl_server_idle = 0;
 		update_curr_dl_se(dl_se->rq, dl_se, delta_exec);
+	}
 }
 
 void dl_server_start(struct sched_dl_entity *dl_se)
@@ -1663,7 +1668,7 @@ void dl_server_start(struct sched_dl_ent
 		setup_new_dl_entity(dl_se);
 	}
 
-	if (!dl_se->dl_runtime)
+	if (!dl_se->dl_runtime || dl_se->dl_server_active)
 		return;
 
 	dl_se->dl_server_active = 1;
@@ -1672,7 +1677,7 @@ void dl_server_start(struct sched_dl_ent
 		resched_curr(dl_se->rq);
 }
 
-void dl_server_stop(struct sched_dl_entity *dl_se)
+static void __dl_server_stop(struct sched_dl_entity *dl_se)
 {
 	if (!dl_se->dl_runtime)
 		return;
@@ -1684,6 +1689,24 @@ void dl_server_stop(struct sched_dl_enti
 	dl_se->dl_server_active = 0;
 }
 
+static bool dl_server_stopped(struct sched_dl_entity *dl_se)
+{
+	if (!dl_se->dl_server_active)
+		return false;
+
+	if (dl_se->dl_server_idle) {
+		__dl_server_stop(dl_se);
+		return true;
+	}
+
+	dl_se->dl_server_idle = 1;
+	return false;
+}
+
+void dl_server_stop(struct sched_dl_entity *dl_se)
+{
+}
+
 void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick_task)
@@ -2435,7 +2458,7 @@ static struct task_struct *__pick_task_d
 	if (dl_server(dl_se)) {
 		p = dl_se->server_pick_task(dl_se);
 		if (!p) {
-			if (dl_server_active(dl_se)) {
+			if (!dl_server_stopped(dl_se)) {
 				dl_se->dl_yielded = 1;
 				update_curr_dl_se(rq, dl_se, 0);
 			}
maintainer
#2 lore raw

Re: [RFC][PATCH 1/5] sched/deadline: Less agressive dl_server handling

Hi,

Quoted 91 lines · Peter Zijlstra

Chris reported that commit 5f6bd380c7bd ("sched/rt: Remove default bandwidth control") caused a significant dip in his favourite benchmark of the day. Simply disabling dl_server cured things.

His workload hammers the 0->1, 1->0 transitions, and the dl_server_{start,stop}() overhead kills it -- fairly obviously a bad idea in hind sight and all that.

Change things around to only disable the dl_server when there has not been a fair task around for a whole period. Since the default period is 1 second, this ensures the benchmark never trips this, overhead gone.

Fixes 557a6bfc662c ("sched/fair: Add trivial fair server")
---
 include/linux/sched.h   |    1 +
 kernel/sched/deadline.c |   31 +++++++++++++++++++++++++++----
 2 files changed, 28 insertions(+), 4 deletions(-)
include/linux/sched.h +1 -0
@@ -702,6 +702,7 @@ struct sched_dl_entity {
 	unsigned int			dl_defer	  : 1;
 	unsigned int			dl_defer_armed	  : 1;
 	unsigned int			dl_defer_running  : 1;
+	unsigned int			dl_server_idle    : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
kernel/sched/deadline.c +15 -3
@@ -1215,6 +1215,8 @@ static void __push_dl_task(struct rq *rq
 /* a defer timer will not be reset if the runtime consumed was < dl_server_min_res */
 static const u64 dl_server_min_res = 1 * NSEC_PER_MSEC;
 
+static bool dl_server_stopped(struct sched_dl_entity *dl_se);
+
 static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = rq_of_dl_se(dl_se);
@@ -1234,6 +1236,7 @@ static enum hrtimer_restart dl_server_ti
 
 		if (!dl_se->server_has_tasks(dl_se)) {
 			replenish_dl_entity(dl_se);
+			dl_server_stopped(dl_se);
 			return HRTIMER_NORESTART;
 		}
 
@@ -1639,8 +1642,10 @@ void dl_server_update_idle_time(struct r
 void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
 {
 	/* 0 runtime = fair server disabled */
-	if (dl_se->dl_runtime)
+	if (dl_se->dl_runtime) {
+		dl_se->dl_server_idle = 0;
 		update_curr_dl_se(dl_se->rq, dl_se, delta_exec);
+	}
 }
 
 void dl_server_start(struct sched_dl_entity *dl_se)
@@ -1663,7 +1668,7 @@ void dl_server_start(struct sched_dl_ent
 		setup_new_dl_entity(dl_se);
 	}
 
-	if (!dl_se->dl_runtime)
+	if (!dl_se->dl_runtime || dl_se->dl_server_active)
 		return;
 
 	dl_se->dl_server_active = 1;
@@ -1672,7 +1677,7 @@ void dl_server_start(struct sched_dl_ent
 		resched_curr(dl_se->rq);
 }
 
-void dl_server_stop(struct sched_dl_entity *dl_se)
+static void __dl_server_stop(struct sched_dl_entity *dl_se)
 {
 	if (!dl_se->dl_runtime)
 		return;
@@ -1684,6 +1689,24 @@ void dl_server_stop(struct sched_dl_enti
 	dl_se->dl_server_active = 0;
 }
 
+static bool dl_server_stopped(struct sched_dl_entity *dl_se)
+{
+	if (!dl_se->dl_server_active)
+		return false;
+
+	if (dl_se->dl_server_idle) {
+		__dl_server_stop(dl_se);

kernel/sched/deadline.c void dl_server_stop(struct sched_dl_enti · line 1708

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+		return true;
+	}
+
+	dl_se->dl_server_idle = 1;
+	return false;
+}
+
+void dl_server_stop(struct sched_dl_entity *dl_se)
+{
+}

What if we explicitly set the server to idle (instead of ignoring the stop) where this gets called in dequeue_entities()? Also, don't we need to actually stop the server if we are changing its parameters from sched_fair_server_write()?

Thanks,
Juri

author maintainer
#3 lore raw

Re: [RFC][PATCH 1/5] sched/deadline: Less agressive dl_server handling

Quoted 19 lines · Juri Lelli

Hi,

Quoted 16 lines
kernel/sched/deadline.c +12 -0
@@ -1684,6 +1689,24 @@ void dl_server_stop(struct sched_dl_enti
 	dl_se->dl_server_active = 0;
 }
 
+static bool dl_server_stopped(struct sched_dl_entity *dl_se)
+{
+	if (!dl_se->dl_server_active)
+		return false;
+
+	if (dl_se->dl_server_idle) {
+		__dl_server_stop(dl_se);
+		return true;
+	}
+
+	dl_se->dl_server_idle = 1;
+	return false;
Quoted 8 lines

kernel/sched/deadline.c void dl_server_stop(struct sched_dl_enti · line 1708

Quoted 5 lines
kernel/sched/deadline.c +5 -0
+}
+
+void dl_server_stop(struct sched_dl_entity *dl_se)
+{
+}

What if we explicitly set the server to idle (instead of ignoring the stop) where this gets called in dequeue_entities()?

That would break thing; we want to detect if it was ever !idle in the period.

Quoted 2 lines

Also, don't we need to actually stop the server if we are changing its parameters from sched_fair_server_write()?

Quite - let me just remove the offending callsites them.

Would this explain this massive regression 0day reported here? Seems weird.

Anyway, let me go update the patch.

author maintainer
#4 lore raw

RFC PATCH 2/5 sched: Optimize ttwu() / select_task_rq()

In reply to Peter Zijlstra

Optimize ttwu() by pushing select_idle_siblings() up above waiting for on_cpu(). This allows making use of the cycles otherwise spend waiting to search for an idle CPU.

One little detail is that since the task we're looking for an idle CPU for might still be on the CPU, that CPU won't report as running the idle task, and thus won't find his own CPU idle, even when it is.

To compensate, remove the 'rq->curr == rq->idle' condition from idle_cpu() -- it doesn't really make sense anyway.

Additionally, Chris found (concurrently) that perf-c2c reported that test as being a cache-miss monster.

---
 kernel/sched/core.c     |    3 ++-
 kernel/sched/syscalls.c |    3 ---
 2 files changed, 2 insertions(+), 4 deletions(-)
kernel/sched/core.c +2 -1
@@ -4305,6 +4305,8 @@ int try_to_wake_up(struct task_struct *p
 		    ttwu_queue_wakelist(p, task_cpu(p), wake_flags))
 			break;
 
+		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
+
 		/*
 		 * If the owning (remote) CPU is still in the middle of schedule() with
 		 * this task as prev, wait until it's done referencing the task.
@@ -4316,7 +4318,6 @@ int try_to_wake_up(struct task_struct *p
 		 */
 		smp_cond_load_acquire(&p->on_cpu, !VAL);
 
-		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
 		if (task_cpu(p) != cpu) {
 			if (p->in_iowait) {
 				delayacct_blkio_end(p);
kernel/sched/syscalls.c +0 -3
@@ -203,9 +203,6 @@ int idle_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
 
-	if (rq->curr != rq->idle)
-		return 0;
-
 	if (rq->nr_running)
 		return 0;
 
#5 lore raw

Re: [RFC][PATCH 2/5] sched: Optimize ttwu() / select_task_rq()

Greetings,

This patch gives RT builds terminal heartburn. This particular boot survived long/well enough to trigger it with LTP sched tests and still be able to crash dump the hung box.

(is_migration_disabled() confirmation thingy below gripe)

[   44.379563] WARNING: CPU: 6 PID: 4468 at kernel/sched/core.c:3354 set_task_cpu+0x1c1/0x1d0
[   44.379569] Modules linked in: af_packet nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security bridge stp llc iscsi_ibft iscsi_boot_sysfs rfkill ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_codec_realtek nls_iso8859_1 nls_cp437 snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_scodec_component snd_hda_intel at24 iTCO_wdt snd_intel_dspcfg r8169 regmap_i2c snd_intel_sdw_acpi intel_pmc_bxt kvm mei_pxp mei_hdcp iTCO_vendor_support realtek snd_hda_codec i2c_i801 mdio_devres snd_hda_core ums_realtek libphy i2c_mux irqbypass pcspkr i2c_smbus snd_hwdep snd_pcm usblp mdio_bus mei_me lpc_ich mfd_core mei snd_timer
[   44.379621]  snd soundcore thermal fan joydev intel_smartconnect tiny_power_button nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc fuse configfs dmi_sysfs ip_tables x_tables uas usb_storage hid_logitech_hidpp hid_logitech_dj hid_generic usbhid nouveau drm_ttm_helper ttm gpu_sched xhci_pci i2c_algo_bit xhci_hcd ahci drm_gpuvm ehci_pci ehci_hcd libahci drm_exec mxm_wmi libata polyval_clmulni usbcore ghash_clmulni_intel drm_display_helper sha512_ssse3 sha1_ssse3 cec rc_core video wmi button sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg scsi_mod scsi_common vfat fat ext4 crc16 mbcache jbd2 loop msr efivarfs aesni_intel
[   44.379663] CPU: 6 UID: 0 PID: 4468 Comm: sandbox_ipc_thr Kdump: loaded Not tainted 6.15.0.ge271ed52-master-rt #19 PREEMPT_{RT,(lazy)}  e4f2516a9b85ac19222adb94a538ef0c57343c1c
[   44.379666] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[   44.379668] RIP: 0010:set_task_cpu+0x1c1/0x1d0
[   44.379670] Code: 0f 0b e9 8f fe ff ff 80 8b 8c 05 00 00 04 e9 f5 fe ff ff 0f 0b e9 7c fe ff ff 0f 0b 66 83 bb 40 04 00 00 00 0f 84 8b fe ff ff <0f> 0b e9 84 fe ff ff 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90
[   44.379672] RSP: 0018:ffffcd844ef77668 EFLAGS: 00010002
[   44.379673] RAX: 0000000000000200 RBX: ffff896da1e8c700 RCX: 0000000000000000
[   44.379675] RDX: ffff896da1e8cb30 RSI: 0000000000000000 RDI: ffff896da1e8c700
[   44.379676] RBP: 0000000000000000 R08: 0000000000000206 R09: 000000000002361d
[   44.379677] R10: fbfffffffffff79d R11: 0000000000000004 R12: 0000000000000000
[   44.379678] R13: 0000000000000000 R14: 0000000000000028 R15: ffff896da1e8d030
[   44.379679] FS:  0000000000000000(0000) GS:ffff8970f15d8000(0000) knlGS:0000000000000000
[   44.379681] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   44.379682] CR2: 00007f4c297c4688 CR3: 000000011b9de002 CR4: 00000000001726f0
[   44.379683] Call Trace:
[   44.379686]  <TASK>
[   44.379688]  try_to_wake_up+0x245/0x810
[   44.379692]  rt_mutex_slowunlock+0x1d2/0x2d0
[   44.379696]  ? __pfx_lru_activate+0x10/0x10
[   44.379700]  folio_batch_move_lru+0xc7/0x100
[   44.379704]  ? __pfx_lru_activate+0x10/0x10
[   44.379706]  __folio_batch_add_and_move+0xf2/0x110
[   44.379710]  folio_mark_accessed+0x80/0x1b0
[   44.379711]  unmap_page_range+0x176b/0x1a60
[   44.379717]  unmap_vmas+0xae/0x1a0
[   44.379720]  exit_mmap+0xe5/0x3c0
[   44.379725]  mmput+0x6e/0x150
[   44.379729]  do_exit+0x23c/0xa20
[   44.379732]  do_group_exit+0x33/0x90
[   44.379735]  get_signal+0x85d/0x8b0
[   44.379738]  arch_do_signal_or_restart+0x2d/0x240
[   44.379743]  ? place_entity+0x1b/0x130
[   44.379745]  ? __x64_sys_poll+0x47/0x1a0
[   44.379749]  exit_to_user_mode_loop+0x86/0x150
[   44.379753]  do_syscall_64+0x1ba/0x8e0
[   44.379756]  ? wakeup_preempt+0x40/0x70
[   44.379758]  ? ttwu_do_activate+0x84/0x210
[   44.379760]  ? _raw_spin_unlock_irqrestore+0x22/0x40
[   44.379763]  ? try_to_wake_up+0xab/0x810
[   44.379765]  ? preempt_count_add+0x4b/0xa0
[   44.379768]  ? futex_hash_put+0x43/0x90
[   44.379772]  ? futex_wake+0xb2/0x1c0
[   44.379775]  ? do_futex+0x125/0x190
[   44.379776]  ? __x64_sys_futex+0x10b/0x1c0
[   44.379779]  ? do_syscall_64+0x7f/0x8e0
[   44.379781]  ? __do_sys_prctl+0xbe/0xee0
[   44.379783]  ? do_syscall_64+0x7f/0x8e0
[   44.379786]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   44.379789] RIP: 0033:0x7f4c3571fdef
[   44.379803] Code: Unable to access opcode bytes at 0x7f4c3571fdc5.
[   44.379803] RSP: 002b:00007f4c0c9fe700 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[   44.379805] RAX: fffffffffffffdfc RBX: 00007f4c0c9fe730 RCX: 00007f4c3571fdef
[   44.379806] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007f4c0c9fe730
[   44.379807] RBP: 00007f4c0c9fe920 R08: 0000000000000000 R09: 0000000000000007
[   44.379808] R10: 00005587485bf1d0 R11: 0000000000000293 R12: 00005587485a7fc0
[   44.379809] R13: 0000000000000000 R14: 0000000000001174 R15: 00007f4c0c1ff000
[   44.379812]  </TASK>
---
 kernel/sched/core.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/core.c +7 -1
@@ -4313,7 +4313,10 @@ int try_to_wake_up(struct task_struct *p
 		    ttwu_queue_wakelist(p, task_cpu(p), wake_flags))
 			break;
 
-		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
+		if (is_migration_disabled(p))
+			cpu = -1;
+		else
+			cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
 
 		/*
 		 * If the owning (remote) CPU is still in the middle of schedule() with
@@ -4326,6 +4329,9 @@ int try_to_wake_up(struct task_struct *p
 		 */
 		smp_cond_load_acquire(&p->on_cpu, !VAL);
 
+		if (cpu == -1)
+			cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
+
 		if (task_cpu(p) != cpu) {
 			if (p->in_iowait) {
 				delayacct_blkio_end(p);
author maintainer
#6 lore raw

Re: [RFC][PATCH 2/5] sched: Optimize ttwu() / select_task_rq()

On Mon, Jun 09, 2025 at 07:01:47AM +0200, Mike Galbraith wrote:

Right; so the problem being that we can race with
migrate_disable_switch().

Quoted 19 lines
 kernel/sched/core.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/core.c +4 -1
@@ -4313,7 +4313,10 @@ int try_to_wake_up(struct task_struct *p
 		    ttwu_queue_wakelist(p, task_cpu(p), wake_flags))
 			break;
 
-		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
+		if (is_migration_disabled(p))
+			cpu = -1;
+		else
+			cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
 
 		/*
 		 * If the owning (remote) CPU is still in the middle of schedule() with

kernel/sched/core.c int try_to_wake_up(struct task_struct *p · line 4337

Quoted 9 lines
kernel/sched/core.c +3 -0
@@ -4326,6 +4329,9 @@ int try_to_wake_up(struct task_struct *p
 		 */
 		smp_cond_load_acquire(&p->on_cpu, !VAL);
 
+		if (cpu == -1)
+			cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
+
 		if (task_cpu(p) != cpu) {
 			if (p->in_iowait) {
 				delayacct_blkio_end(p);

So select_task_rq() already checks is_migration_disabled(); just not well enough. Also, I'm thinking that if we see migration_disabled, we don't need to call it a second time, just let it be where it was.

Does something like this help? Specifically, when nr_cpus_allowed == 1 || is_migration_disabled(), don't change @cpu at all.

kernel/sched/core.c +1 -1
@@ -3593,7 +3593,7 @@ int select_task_rq(struct task_struct *p
 		cpu = p->sched_class->select_task_rq(p, cpu, *wake_flags);
 		*wake_flags |= WF_RQ_SELECTED;
 	} else {
-		cpu = cpumask_any(p->cpus_ptr);
+		cpu = task_cpu(p);
 	}
 
 	/*
#7 lore raw

Re: [RFC][PATCH 2/5] sched: Optimize ttwu() / select_task_rq()

Quoted 4 lines · Peter Zijlstra

On Mon, Jun 09, 2025 at 07:01:47AM +0200, Mike Galbraith wrote:

Right; so the problem being that we can race with
migrate_disable_switch().

Yeah. Most of the time when we do fallback saves us, but we can and do zip past it, and that turns box various shades of sad.

Quoted 2 lines

Does something like this help?

It surely will, but I'll testdrive it. No news is good news.

Quoted 12 lines
kernel/sched/core.c +1 -1
@@ -3593,7 +3593,7 @@ int select_task_rq(struct task_struct *p
  		cpu = p->sched_class->select_task_rq(p, cpu,
 *wake_flags);
  		*wake_flags |= WF_RQ_SELECTED;
  	} else {
-		cpu = cpumask_any(p->cpus_ptr);
+		cpu = task_cpu(p);
  	}


  /*

author maintainer
#8 lore raw

RFC PATCH 3/5 sched: Split up ttwu_runnable()

In reply to Peter Zijlstra

Split up ttwu_runnable() in preparation for more changes.

---
 kernel/sched/core.c  |   43 +++++++++++++++++++++----------------------
 kernel/sched/sched.h |    5 +++++
 2 files changed, 26 insertions(+), 22 deletions(-)
kernel/sched/core.c +21 -22
@@ -3765,6 +3765,25 @@ ttwu_do_activate(struct rq *rq, struct t
 #endif
 }
 
+static int __ttwu_runnable(struct rq *rq, struct task_struct *p, int wake_flags)
+{
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed)
+		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
+	}
+	ttwu_do_wakeup(p);
+	return 1;
+}
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3792,28 +3811,8 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
-		}
-		ttwu_do_wakeup(p);
-		ret = 1;
-	}
-	__task_rq_unlock(rq, &rf);
-
-	return ret;
+	CLASS(__task_rq_lock, guard)(p);
+	return __ttwu_runnable(guard.rq, p, wake_flags);
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1831,6 +1831,11 @@ task_rq_unlock(struct rq *rq, struct tas
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+		    _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+		    __task_rq_unlock(_T->rq, &_T->rf),
+		    struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
 		    _T->rq = task_rq_lock(_T->lock, &_T->rf),
 		    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
author maintainer
#9 lore raw

RFC PATCH 4/5 sched: Add ttwu_queue controls

In reply to Peter Zijlstra

There are two (soon three) callers of ttwu_queue_wakelist(), distinguish them with their own WF_ and add some knobs on.

---
 kernel/sched/core.c     |   22 ++++++++++++----------
 kernel/sched/features.h |    2 ++
 kernel/sched/sched.h    |    2 ++
 3 files changed, 16 insertions(+), 10 deletions(-)
kernel/sched/core.c +12 -10
@@ -3928,7 +3928,7 @@ bool cpus_share_resources(int this_cpu,
 	return per_cpu(sd_share_id, this_cpu) == per_cpu(sd_share_id, that_cpu);
 }
 
-static inline bool ttwu_queue_cond(struct task_struct *p, int cpu)
+static inline bool ttwu_queue_cond(struct task_struct *p, int cpu, bool def)
 {
 	/* See SCX_OPS_ALLOW_QUEUED_WAKEUP. */
 	if (!scx_allow_ttwu_queue(p))
@@ -3969,18 +3969,19 @@ static inline bool ttwu_queue_cond(struc
 	if (!cpu_rq(cpu)->nr_running)
 		return true;
 
-	return false;
+	return def;
 }
 
 static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags)
 {
-	if (sched_feat(TTWU_QUEUE) && ttwu_queue_cond(p, cpu)) {
-		sched_clock_cpu(cpu); /* Sync clocks across CPUs */
-		__ttwu_queue_wakelist(p, cpu, wake_flags);
-		return true;
-	}
+	bool def = sched_feat(TTWU_QUEUE_DEFAULT);
 
-	return false;
+	if (!ttwu_queue_cond(p, cpu, def))
+		return false;
+
+	sched_clock_cpu(cpu); /* Sync clocks across CPUs */
+	__ttwu_queue_wakelist(p, cpu, wake_flags);
+	return true;
 }
 
 #else /* !CONFIG_SMP */
@@ -3997,7 +3998,7 @@ static void ttwu_queue(struct task_struc
 	struct rq *rq = cpu_rq(cpu);
 	struct rq_flags rf;
 
-	if (ttwu_queue_wakelist(p, cpu, wake_flags))
+	if (sched_feat(TTWU_QUEUE) && ttwu_queue_wakelist(p, cpu, wake_flags))
 		return;
 
 	rq_lock(rq, &rf);
@@ -4301,7 +4302,8 @@ int try_to_wake_up(struct task_struct *p
 		 * scheduling.
 		 */
 		if (smp_load_acquire(&p->on_cpu) &&
-		    ttwu_queue_wakelist(p, task_cpu(p), wake_flags))
+		    sched_feat(TTWU_QUEUE_ON_CPU) &&
+		    ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_ON_CPU))
 			break;
 
 		cpu = select_task_rq(p, p->wake_cpu, &wake_flags);
kernel/sched/features.h +2 -0
@@ -81,6 +81,8 @@ SCHED_FEAT(TTWU_QUEUE, false)
  */
 SCHED_FEAT(TTWU_QUEUE, true)
 #endif
+SCHED_FEAT(TTWU_QUEUE_ON_CPU, true)
+SCHED_FEAT(TTWU_QUEUE_DEFAULT, false)
 
 /*
  * When doing wakeups, attempt to limit superfluous scans of the LLC domain.
kernel/sched/sched.h +2 -0
@@ -2312,6 +2312,8 @@ static inline int task_on_rq_migrating(s
 #define WF_CURRENT_CPU		0x40 /* Prefer to move the wakee to the current CPU. */
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
 
+#define WF_ON_CPU		0x0100
+
 #ifdef CONFIG_SMP
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
 static_assert(WF_FORK == SD_BALANCE_FORK);
author maintainer
#10 lore raw

RFC PATCH 5/5 sched: Add ttwu_queue support for delayed tasks

In reply to Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

Doing the in-place thing adds quite a bit of cross-cpu latency, add a little something that gets remote CPUs to do their own in-place wakeups, significantly reducing the rq->lock contention.

---
 kernel/sched/core.c     |   74 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/fair.c     |    5 ++-
 kernel/sched/features.h |    1 
 kernel/sched/sched.h    |    1 
 4 files changed, 72 insertions(+), 9 deletions(-)
kernel/sched/core.c +66 -8
@@ -3784,6 +3784,8 @@ static int __ttwu_runnable(struct rq *rq
 	return 1;
 }
 
+static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags);
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3811,6 +3813,33 @@ static int __ttwu_runnable(struct rq *rq
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
+#ifdef CONFIG_SMP
+	if (sched_feat(TTWU_QUEUE_DELAYED) && READ_ONCE(p->se.sched_delayed)) {
+		/*
+		 * Similar to try_to_block_task():
+		 *
+		 * __schedule()				ttwu()
+		 *   prev_state = prev->state		  if (p->sched_delayed)
+		 *   if (prev_state)			     smp_acquire__after_ctrl_dep()
+		 *     try_to_block_task()		     p->state = TASK_WAKING
+		 *       ... set_delayed()
+		 *         RELEASE p->sched_delayed = 1
+		 *
+		 * __schedule() and ttwu() have matching control dependencies.
+		 *
+		 * Notably, once we observe sched_delayed we know the task has
+		 * passed try_to_block_task() and p->state is ours to modify.
+		 *
+		 * TASK_WAKING controls ttwu() concurrency.
+		 */
+		smp_acquire__after_ctrl_dep();
+		WRITE_ONCE(p->__state, TASK_WAKING);
+
+		if (ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_DELAYED))
+			return 1;
+	}
+#endif
+
 	CLASS(__task_rq_lock, guard)(p);
 	return __ttwu_runnable(guard.rq, p, wake_flags);
 }
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
 	update_rq_clock(rq);
 
 	llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+		struct rq *p_rq = task_rq(p);
+		int ret;
+
+		/*
+		 * This is the ttwu_runnable() case. Notably it is possible for
+		 * on-rq entities to get migrated -- even sched_delayed ones.
+		 */
+		if (unlikely(p_rq != rq)) {
+			rq_unlock(rq, &rf);
+			p_rq = __task_rq_lock(p, &rf);
+		}
+
+		ret = __ttwu_runnable(p_rq, p, WF_TTWU);
+
+		if (unlikely(p_rq != rq)) {
+			if (!ret)
+				set_task_cpu(p, cpu_of(rq));
+
+			__task_rq_unlock(p_rq, &rf);
+			rq_lock(rq, &rf);
+			update_rq_clock(rq);
+		}
+
+		if (ret) {
+			// XXX ttwu_stat()
+			continue;
+		}
+
+		/*
+		 * This is the 'normal' case where the task is blocked.
+		 */
+
 		if (WARN_ON_ONCE(p->on_cpu))
 			smp_cond_load_acquire(&p->on_cpu, !VAL);
 
-		if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq)))
-			set_task_cpu(p, cpu_of(rq));
-
 		ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
 	}
 
@@ -3974,7 +4032,7 @@ static inline bool ttwu_queue_cond(struc
 
 static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags)
 {
-	bool def = sched_feat(TTWU_QUEUE_DEFAULT);
+	bool def = sched_feat(TTWU_QUEUE_DEFAULT) || (wake_flags & WF_DELAYED);
 
 	if (!ttwu_queue_cond(p, cpu, def))
 		return false;
@@ -4269,8 +4327,8 @@ int try_to_wake_up(struct task_struct *p
 		 * __schedule().  See the comment for smp_mb__after_spinlock().
 		 *
 		 * Form a control-dep-acquire with p->on_rq == 0 above, to ensure
-		 * schedule()'s deactivate_task() has 'happened' and p will no longer
-		 * care about it's own p->state. See the comment in __schedule().
+		 * schedule()'s try_to_block_task() has 'happened' and p will no longer
+		 * care about it's own p->state. See the comment in try_to_block_task().
 		 */
 		smp_acquire__after_ctrl_dep();
 
@@ -6712,8 +6770,8 @@ static void __sched notrace __schedule(i
 	preempt = sched_mode == SM_PREEMPT;
 
 	/*
-	 * We must load prev->state once (task_struct::state is volatile), such
-	 * that we form a control dependency vs deactivate_task() below.
+	 * We must load prev->state once, such that we form a control
+	 * dependency vs try_to_block_task() below.
 	 */
 	prev_state = READ_ONCE(prev->__state);
 	if (sched_mode == SM_IDLE) {
kernel/sched/fair.c +4 -1
@@ -5395,7 +5395,10 @@ static __always_inline void return_cfs_r
 
 static void set_delayed(struct sched_entity *se)
 {
-	se->sched_delayed = 1;
+	/*
+	 * See TTWU_QUEUE_DELAYED in ttwu_runnable().
+	 */
+	smp_store_release(&se->sched_delayed, 1);
 
 	/*
 	 * Delayed se of cfs_rq have no tasks queued on them.
kernel/sched/features.h +1 -0
@@ -82,6 +82,7 @@ SCHED_FEAT(TTWU_QUEUE, false)
 SCHED_FEAT(TTWU_QUEUE, true)
 #endif
 SCHED_FEAT(TTWU_QUEUE_ON_CPU, true)
+SCHED_FEAT(TTWU_QUEUE_DELAYED, false)
 SCHED_FEAT(TTWU_QUEUE_DEFAULT, false)
 
 /*
kernel/sched/sched.h +1 -0
@@ -2313,6 +2313,7 @@ static inline int task_on_rq_migrating(s
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
 
 #define WF_ON_CPU		0x0100
+#define WF_DELAYED		0x0200
 
 #ifdef CONFIG_SMP
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
maintainer
#11 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC, the task is still enqueued on the same CPU but the target cpu will do the enqueue itself instead on the local CPU. Or am I missing something ?

Quoted 60 lines

Doing the in-place thing adds quite a bit of cross-cpu latency, add a little something that gets remote CPUs to do their own in-place wakeups, significantly reducing the rq->lock contention.

---
 kernel/sched/core.c     |   74 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/fair.c     |    5 ++-
 kernel/sched/features.h |    1
 kernel/sched/sched.h    |    1
 4 files changed, 72 insertions(+), 9 deletions(-)
kernel/sched/core.c +29 -0
@@ -3784,6 +3784,8 @@ static int __ttwu_runnable(struct rq *rq
        return 1;
 }
 
+static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags);
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3811,6 +3813,33 @@ static int __ttwu_runnable(struct rq *rq
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
+#ifdef CONFIG_SMP
+       if (sched_feat(TTWU_QUEUE_DELAYED) && READ_ONCE(p->se.sched_delayed)) {
+               /*
+                * Similar to try_to_block_task():
+                *
+                * __schedule()                         ttwu()
+                *   prev_state = prev->state             if (p->sched_delayed)
+                *   if (prev_state)                         smp_acquire__after_ctrl_dep()
+                *     try_to_block_task()                   p->state = TASK_WAKING
+                *       ... set_delayed()
+                *         RELEASE p->sched_delayed = 1
+                *
+                * __schedule() and ttwu() have matching control dependencies.
+                *
+                * Notably, once we observe sched_delayed we know the task has
+                * passed try_to_block_task() and p->state is ours to modify.
+                *
+                * TASK_WAKING controls ttwu() concurrency.
+                */
+               smp_acquire__after_ctrl_dep();
+               WRITE_ONCE(p->__state, TASK_WAKING);
+
+               if (ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_DELAYED))
+                       return 1;
+       }
+#endif
+
        CLASS(__task_rq_lock, guard)(p);
        return __ttwu_runnable(guard.rq, p, wake_flags);
 }

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 9 lines
kernel/sched/core.c +6 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Quoted 83 lines
kernel/sched/core.c +31 -8
+                */
+               if (unlikely(p_rq != rq)) {
+                       rq_unlock(rq, &rf);
+                       p_rq = __task_rq_lock(p, &rf);
+               }
+
+               ret = __ttwu_runnable(p_rq, p, WF_TTWU);
+
+               if (unlikely(p_rq != rq)) {
+                       if (!ret)
+                               set_task_cpu(p, cpu_of(rq));
+
+                       __task_rq_unlock(p_rq, &rf);
+                       rq_lock(rq, &rf);
+                       update_rq_clock(rq);
+               }
+
+               if (ret) {
+                       // XXX ttwu_stat()
+                       continue;
+               }
+
+               /*
+                * This is the 'normal' case where the task is blocked.
+                */
+
                if (WARN_ON_ONCE(p->on_cpu))
                        smp_cond_load_acquire(&p->on_cpu, !VAL);
 
-               if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq)))
-                       set_task_cpu(p, cpu_of(rq));
-
                ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
        }
 
@@ -3974,7 +4032,7 @@ static inline bool ttwu_queue_cond(struc
 
 static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags)
 {
-       bool def = sched_feat(TTWU_QUEUE_DEFAULT);
+       bool def = sched_feat(TTWU_QUEUE_DEFAULT) || (wake_flags & WF_DELAYED);
 
        if (!ttwu_queue_cond(p, cpu, def))
                return false;
@@ -4269,8 +4327,8 @@ int try_to_wake_up(struct task_struct *p
                 * __schedule().  See the comment for smp_mb__after_spinlock().
                 *
                 * Form a control-dep-acquire with p->on_rq == 0 above, to ensure
-                * schedule()'s deactivate_task() has 'happened' and p will no longer
-                * care about it's own p->state. See the comment in __schedule().
+                * schedule()'s try_to_block_task() has 'happened' and p will no longer
+                * care about it's own p->state. See the comment in try_to_block_task().
                 */
                smp_acquire__after_ctrl_dep();
 
@@ -6712,8 +6770,8 @@ static void __sched notrace __schedule(i
        preempt = sched_mode == SM_PREEMPT;
 
        /*
-        * We must load prev->state once (task_struct::state is volatile), such
-        * that we form a control dependency vs deactivate_task() below.
+        * We must load prev->state once, such that we form a control
+        * dependency vs try_to_block_task() below.
         */
        prev_state = READ_ONCE(prev->__state);
        if (sched_mode == SM_IDLE) {
kernel/sched/fair.c +4 -1
@@ -5395,7 +5395,10 @@ static __always_inline void return_cfs_r
 
 static void set_delayed(struct sched_entity *se)
 {
-       se->sched_delayed = 1;
+       /*
+        * See TTWU_QUEUE_DELAYED in ttwu_runnable().
+        */
+       smp_store_release(&se->sched_delayed, 1);
 
        /*
         * Delayed se of cfs_rq have no tasks queued on them.

kernel/sched/features.h SCHED_FEAT(TTWU_QUEUE, false) · line 85

Quoted 4 lines
kernel/sched/features.h +1 -0
@@ -82,6 +82,7 @@ SCHED_FEAT(TTWU_QUEUE, false)
 SCHED_FEAT(TTWU_QUEUE, true)
 #endif
 SCHED_FEAT(TTWU_QUEUE_ON_CPU, true)
+SCHED_FEAT(TTWU_QUEUE_DELAYED, false)

I'm not sure that the feature will be tested as people mainly test default config

Quoted 15 lines
kernel/sched/features.h +0 -0
 SCHED_FEAT(TTWU_QUEUE_DEFAULT, false)
 
 /*
kernel/sched/sched.h +1 -0
@@ -2313,6 +2313,7 @@ static inline int task_on_rq_migrating(s
 #define WF_RQ_SELECTED         0x80 /* ->select_task_rq() was called */
 
 #define WF_ON_CPU              0x0100
+#define WF_DELAYED             0x0200
 
 #ifdef CONFIG_SMP
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
author maintainer
#12 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 6 lines · Vincent Guittot
Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

Quoted 6 lines

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC, the task is still enqueued on the same CPU but the target cpu will do the enqueue itself instead on the local CPU. Or am I missing something ?

Correct. I tried to add that migration into the mix, but then things get tricky real fast.

Just getting rid of the remote rq lock also helped; these dispatch threads just need to get on with waking up tasks, any delay hurts.

Quoted 62 lines
Quoted 62 lines

Doing the in-place thing adds quite a bit of cross-cpu latency, add a little something that gets remote CPUs to do their own in-place wakeups, significantly reducing the rq->lock contention.

---
 kernel/sched/core.c     |   74 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/fair.c     |    5 ++-
 kernel/sched/features.h |    1
 kernel/sched/sched.h    |    1
 4 files changed, 72 insertions(+), 9 deletions(-)
kernel/sched/core.c +29 -0
@@ -3784,6 +3784,8 @@ static int __ttwu_runnable(struct rq *rq
        return 1;
 }
 
+static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags);
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3811,6 +3813,33 @@ static int __ttwu_runnable(struct rq *rq
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
+#ifdef CONFIG_SMP
+       if (sched_feat(TTWU_QUEUE_DELAYED) && READ_ONCE(p->se.sched_delayed)) {
+               /*
+                * Similar to try_to_block_task():
+                *
+                * __schedule()                         ttwu()
+                *   prev_state = prev->state             if (p->sched_delayed)
+                *   if (prev_state)                         smp_acquire__after_ctrl_dep()
+                *     try_to_block_task()                   p->state = TASK_WAKING
+                *       ... set_delayed()
+                *         RELEASE p->sched_delayed = 1
+                *
+                * __schedule() and ttwu() have matching control dependencies.
+                *
+                * Notably, once we observe sched_delayed we know the task has
+                * passed try_to_block_task() and p->state is ours to modify.
+                *
+                * TASK_WAKING controls ttwu() concurrency.
+                */
+               smp_acquire__after_ctrl_dep();
+               WRITE_ONCE(p->__state, TASK_WAKING);
+
+               if (ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_DELAYED))
+                       return 1;
+       }
+#endif
+
        CLASS(__task_rq_lock, guard)(p);
        return __ttwu_runnable(guard.rq, p, wake_flags);
 }
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
Quoted 9 lines

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 7 lines
kernel/sched/core.c +6 -0
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Doesn't happen often, but it can happen. Nothing really stops it from happening. Eg weight based balancing can do it. As can numa balancing and affinity changes.

maintainer
#13 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 15 lines · Peter Zijlstra
Quoted 6 lines · Vincent Guittot
Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

Quoted 6 lines

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC, the task is still enqueued on the same CPU but the target cpu will do the enqueue itself instead on the local CPU. Or am I missing something ?

Quoted 2 lines

Correct. I tried to add that migration into the mix, but then things get tricky real fast.

Yeah, I can imagine

Quoted 76 lines

Just getting rid of the remote rq lock also helped; these dispatch threads just need to get on with waking up tasks, any delay hurts.

Quoted 62 lines
Quoted 62 lines

Doing the in-place thing adds quite a bit of cross-cpu latency, add a little something that gets remote CPUs to do their own in-place wakeups, significantly reducing the rq->lock contention.

---
 kernel/sched/core.c     |   74 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/fair.c     |    5 ++-
 kernel/sched/features.h |    1
 kernel/sched/sched.h    |    1
 4 files changed, 72 insertions(+), 9 deletions(-)
kernel/sched/core.c +29 -0
@@ -3784,6 +3784,8 @@ static int __ttwu_runnable(struct rq *rq
        return 1;
 }
 
+static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags);
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3811,6 +3813,33 @@ static int __ttwu_runnable(struct rq *rq
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
+#ifdef CONFIG_SMP
+       if (sched_feat(TTWU_QUEUE_DELAYED) && READ_ONCE(p->se.sched_delayed)) {
+               /*
+                * Similar to try_to_block_task():
+                *
+                * __schedule()                         ttwu()
+                *   prev_state = prev->state             if (p->sched_delayed)
+                *   if (prev_state)                         smp_acquire__after_ctrl_dep()
+                *     try_to_block_task()                   p->state = TASK_WAKING
+                *       ... set_delayed()
+                *         RELEASE p->sched_delayed = 1
+                *
+                * __schedule() and ttwu() have matching control dependencies.
+                *
+                * Notably, once we observe sched_delayed we know the task has
+                * passed try_to_block_task() and p->state is ours to modify.
+                *
+                * TASK_WAKING controls ttwu() concurrency.
+                */
+               smp_acquire__after_ctrl_dep();
+               WRITE_ONCE(p->__state, TASK_WAKING);
+
+               if (ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_DELAYED))
+                       return 1;
+       }
+#endif
+
        CLASS(__task_rq_lock, guard)(p);
        return __ttwu_runnable(guard.rq, p, wake_flags);
 }
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
Quoted 9 lines

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 7 lines
kernel/sched/core.c +6 -0
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Quoted 3 lines

Doesn't happen often, but it can happen. Nothing really stops it from happening. Eg weight based balancing can do it. As can numa balancing and affinity changes.

Yes, I agree that delayed tasks can migrate because of load balancing but not at wake up.

author maintainer
#14 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 13 lines · Vincent Guittot
Quoted 13 lines
Quoted 12 lines

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 10 lines
kernel/sched/core.c +6 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Quoted 6 lines
Quoted 3 lines

Doesn't happen often, but it can happen. Nothing really stops it from happening. Eg weight based balancing can do it. As can numa balancing and affinity changes.

Yes, I agree that delayed tasks can migrate because of load balancing but not at wake up.

Right, but this here is the case where wakeup races with load-balancing. Specifically, due to the wake_list, the wakeup can happen while the task is on CPU N, and by the time the IPI gets processed the task has moved to CPU M.

It doesn't happen often, but it was 'fun' chasing that fail around for a day :/

maintainer
#15 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 22 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
Quoted 13 lines
Quoted 12 lines

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 10 lines
kernel/sched/core.c +6 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Quoted 6 lines
Quoted 3 lines

Doesn't happen often, but it can happen. Nothing really stops it from happening. Eg weight based balancing can do it. As can numa balancing and affinity changes.

Yes, I agree that delayed tasks can migrate because of load balancing but not at wake up.

Quoted 7 lines

Right, but this here is the case where wakeup races with load-balancing. Specifically, due to the wake_list, the wakeup can happen while the task is on CPU N, and by the time the IPI gets processed the task has moved to CPU M.

It doesn't happen often, but it was 'fun' chasing that fail around for a day :/

Ok, it makes sense now.

#16 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

In reply to Vincent Guittot

Hi Peter,

Quoted 6 lines · On Fri, Jun 06, 2025 at 05:03:36PM +0200 Vincent Guittot wrote:
Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

Quoted 6 lines

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC, the task is still enqueued on the same CPU but the target cpu will do the enqueue itself instead on the local CPU. Or am I missing something ?

Yeah, this one still bites us. We ran these patches on our perf tests (with out twiddling any FEATs) and it was basically a wash.

The fs regression we saw due to always waking up on the same cpu was still present as expected based on this patch I suppose.

Thanks,
Phil

Quoted 180 lines
Quoted 61 lines

Doing the in-place thing adds quite a bit of cross-cpu latency, add a little something that gets remote CPUs to do their own in-place wakeups, significantly reducing the rq->lock contention.

---
 kernel/sched/core.c     |   74 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/fair.c     |    5 ++-
 kernel/sched/features.h |    1
 kernel/sched/sched.h    |    1
 4 files changed, 72 insertions(+), 9 deletions(-)
kernel/sched/core.c +29 -0
@@ -3784,6 +3784,8 @@ static int __ttwu_runnable(struct rq *rq
        return 1;
 }
 
+static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags);
+
 /*
  * Consider @p being inside a wait loop:
  *
@@ -3811,6 +3813,33 @@ static int __ttwu_runnable(struct rq *rq
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
+#ifdef CONFIG_SMP
+       if (sched_feat(TTWU_QUEUE_DELAYED) && READ_ONCE(p->se.sched_delayed)) {
+               /*
+                * Similar to try_to_block_task():
+                *
+                * __schedule()                         ttwu()
+                *   prev_state = prev->state             if (p->sched_delayed)
+                *   if (prev_state)                         smp_acquire__after_ctrl_dep()
+                *     try_to_block_task()                   p->state = TASK_WAKING
+                *       ... set_delayed()
+                *         RELEASE p->sched_delayed = 1
+                *
+                * __schedule() and ttwu() have matching control dependencies.
+                *
+                * Notably, once we observe sched_delayed we know the task has
+                * passed try_to_block_task() and p->state is ours to modify.
+                *
+                * TASK_WAKING controls ttwu() concurrency.
+                */
+               smp_acquire__after_ctrl_dep();
+               WRITE_ONCE(p->__state, TASK_WAKING);
+
+               if (ttwu_queue_wakelist(p, task_cpu(p), wake_flags | WF_DELAYED))
+                       return 1;
+       }
+#endif
+
        CLASS(__task_rq_lock, guard)(p);
        return __ttwu_runnable(guard.rq, p, wake_flags);
 }

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3867

Quoted 9 lines
kernel/sched/core.c +6 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
        update_rq_clock(rq);
 
        llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {
+               struct rq *p_rq = task_rq(p);
+               int ret;
+
+               /*
+                * This is the ttwu_runnable() case. Notably it is possible for
+                * on-rq entities to get migrated -- even sched_delayed ones.

I haven't found where the sched_delayed task could migrate on another cpu.

Quoted 83 lines
kernel/sched/core.c +31 -8
+                */
+               if (unlikely(p_rq != rq)) {
+                       rq_unlock(rq, &rf);
+                       p_rq = __task_rq_lock(p, &rf);
+               }
+
+               ret = __ttwu_runnable(p_rq, p, WF_TTWU);
+
+               if (unlikely(p_rq != rq)) {
+                       if (!ret)
+                               set_task_cpu(p, cpu_of(rq));
+
+                       __task_rq_unlock(p_rq, &rf);
+                       rq_lock(rq, &rf);
+                       update_rq_clock(rq);
+               }
+
+               if (ret) {
+                       // XXX ttwu_stat()
+                       continue;
+               }
+
+               /*
+                * This is the 'normal' case where the task is blocked.
+                */
+
                if (WARN_ON_ONCE(p->on_cpu))
                        smp_cond_load_acquire(&p->on_cpu, !VAL);
 
-               if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq)))
-                       set_task_cpu(p, cpu_of(rq));
-
                ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
        }
 
@@ -3974,7 +4032,7 @@ static inline bool ttwu_queue_cond(struc
 
 static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags)
 {
-       bool def = sched_feat(TTWU_QUEUE_DEFAULT);
+       bool def = sched_feat(TTWU_QUEUE_DEFAULT) || (wake_flags & WF_DELAYED);
 
        if (!ttwu_queue_cond(p, cpu, def))
                return false;
@@ -4269,8 +4327,8 @@ int try_to_wake_up(struct task_struct *p
                 * __schedule().  See the comment for smp_mb__after_spinlock().
                 *
                 * Form a control-dep-acquire with p->on_rq == 0 above, to ensure
-                * schedule()'s deactivate_task() has 'happened' and p will no longer
-                * care about it's own p->state. See the comment in __schedule().
+                * schedule()'s try_to_block_task() has 'happened' and p will no longer
+                * care about it's own p->state. See the comment in try_to_block_task().
                 */
                smp_acquire__after_ctrl_dep();
 
@@ -6712,8 +6770,8 @@ static void __sched notrace __schedule(i
        preempt = sched_mode == SM_PREEMPT;
 
        /*
-        * We must load prev->state once (task_struct::state is volatile), such
-        * that we form a control dependency vs deactivate_task() below.
+        * We must load prev->state once, such that we form a control
+        * dependency vs try_to_block_task() below.
         */
        prev_state = READ_ONCE(prev->__state);
        if (sched_mode == SM_IDLE) {
kernel/sched/fair.c +4 -1
@@ -5395,7 +5395,10 @@ static __always_inline void return_cfs_r
 
 static void set_delayed(struct sched_entity *se)
 {
-       se->sched_delayed = 1;
+       /*
+        * See TTWU_QUEUE_DELAYED in ttwu_runnable().
+        */
+       smp_store_release(&se->sched_delayed, 1);
 
        /*
         * Delayed se of cfs_rq have no tasks queued on them.

kernel/sched/features.h SCHED_FEAT(TTWU_QUEUE, false) · line 85

Quoted 4 lines
kernel/sched/features.h +1 -0
@@ -82,6 +82,7 @@ SCHED_FEAT(TTWU_QUEUE, false)
 SCHED_FEAT(TTWU_QUEUE, true)
 #endif
 SCHED_FEAT(TTWU_QUEUE_ON_CPU, true)
+SCHED_FEAT(TTWU_QUEUE_DELAYED, false)

I'm not sure that the feature will be tested as people mainly test default config

Quoted 15 lines
kernel/sched/features.h +0 -0
 SCHED_FEAT(TTWU_QUEUE_DEFAULT, false)
 
 /*
kernel/sched/sched.h +1 -0
@@ -2313,6 +2313,7 @@ static inline int task_on_rq_migrating(s
 #define WF_RQ_SELECTED         0x80 /* ->select_task_rq() was called */
 
 #define WF_ON_CPU              0x0100
+#define WF_DELAYED             0x0200
 
 #ifdef CONFIG_SMP
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
author maintainer
#17 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

In reply to Vincent Guittot
Quoted 10 lines · Vincent Guittot
Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC,

FWIW, the trivial form of all this is something like the below. The problem is that performance sucks :/ For me it is worse than not doing it. But perhaps it is the right thing for the more complicated cases ?

On my SPR:

schbench-6.9.0-1.txt:average rps: 2975450.75
schbench-6.9.0-2.txt:average rps: 2975464.38
schbench-6.9.0-3.txt:average rps: 2974881.02

(these patches) schbench-6.15.0-dirty-1.txt:average rps: 3029984.58 schbench-6.15.0-dirty-2.txt:average rps: 3034723.10 schbench-6.15.0-dirty-3.txt:average rps: 3033893.33

TTWU_QUEUE_DELAYED schbench-6.15.0-dirty-delayed-1.txt:average rps: 3048778.58 schbench-6.15.0-dirty-delayed-2.txt:average rps: 3049587.90 schbench-6.15.0-dirty-delayed-3.txt:average rps: 3045826.95

NO_DELAY_DEQUEUE schbench-6.15.0-dirty-no_delay-1.txt:average rps: 3043629.03 schbench-6.15.0-dirty-no_delay-2.txt:average rps: 3046054.47 schbench-6.15.0-dirty-no_delay-3.txt:average rps: 3044736.37

TTWU_DEQUEUE schbench-6.15.0-dirty-dequeue-1.txt:average rps: 3008790.80 schbench-6.15.0-dirty-dequeue-2.txt:average rps: 3017497.33 schbench-6.15.0-dirty-dequeue-3.txt:average rps: 3005858.57

kernel/sched/core.c +6 -1
Index: linux-2.6/kernel/sched/core.c
===================================================================
@@ -3770,8 +3770,13 @@ static int __ttwu_runnable(struct rq *rq
 		return 0;
 
 	update_rq_clock(rq);
-	if (p->se.sched_delayed)
+	if (p->se.sched_delayed) {
+		if (sched_feat(TTWU_DEQUEUE)) {
+			dequeue_task(rq, p, DEQUEUE_NOCLOCK | DEQUEUE_DELAYED | DEQUEUE_SLEEP);
+			return 0;
+		}
 		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+	}
 	if (!task_on_cpu(rq, p)) {
 		/*
 		 * When on_rq && !on_cpu the task is preempted, see if
kernel/sched/features.h +1 -0
Index: linux-2.6/kernel/sched/features.h
===================================================================
@@ -84,6 +84,7 @@ SCHED_FEAT(TTWU_QUEUE, true)
 SCHED_FEAT(TTWU_QUEUE_ON_CPU, true)
 SCHED_FEAT(TTWU_QUEUE_DELAYED, false)
 SCHED_FEAT(TTWU_QUEUE_DEFAULT, false)
+SCHED_FEAT(TTWU_DEQUEUE, false)
 
 /*
  * When doing wakeups, attempt to limit superfluous scans of the LLC domain.
author maintainer
#18 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Vincent Guittot
Quoted 4 lines · Peter Zijlstra

One of the things lost with introduction of DELAY_DEQUEUE is the ability of TTWU to move those tasks around on wakeup, since they're on_rq, and as such, need to be woken in-place.

Quoted 8 lines
Quoted 4 lines

I was thinking that you would call select_task_rq() somewhere in the wake up path of delayed entity to get a chance to migrate it which was one reason for the perf regression (and which would have also been useful for EAS case) but IIUC,

FWIW, the trivial form of all this is something like the below. The problem is that performance sucks :/ For me it is worse than not doing it.

And because I was poking at the thing, I had to try the complicated version again... This seems to survive long enough for a few benchmark runs, and its not bad.

It very much burns after a while though :-( So I'll have to poke more at this. Clearly I'm missing something (again!).

---

include/linux/sched.h +1 -0
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -994,6 +994,7 @@ struct task_struct {
 	 * ->sched_remote_wakeup gets used, so it can be in this word.
 	 */
 	unsigned			sched_remote_wakeup:1;
+	unsigned			sched_remote_delayed:1;
 #ifdef CONFIG_RT_MUTEXES
 	unsigned			sched_rt_mutex:1;
 #endif
kernel/sched/core.c +49 -40

117 lines not on this page · raw message

#19 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

In reply to Peter Zijlstra

On 20/05/2025 11:45, Peter Zijlstra wrote:

[...]

Quoted 4 lines
kernel/sched/core.c +0 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
 	update_rq_clock(rq);
 
 	llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3871

Quoted 10 lines
kernel/sched/core.c +10 -0
+		struct rq *p_rq = task_rq(p);
+		int ret;
+
+		/*
+		 * This is the ttwu_runnable() case. Notably it is possible for
+		 * on-rq entities to get migrated -- even sched_delayed ones.
+		 */
+		if (unlikely(p_rq != rq)) {
+			rq_unlock(rq, &rf);
+			p_rq = __task_rq_lock(p, &rf);

I always get this fairly early with TTWU_QUEUE_DELAYED enabled, related to p->pi_lock not held in wakeup from interrupt.

[   36.175285] WARNING: CPU: 0 PID: 162 at kernel/sched/core.c:679 __task_rq_lock+0xf8/0x128
[   36.176021] Modules linked in:
[   36.176187] CPU: 0 UID: 0 PID: 162 Comm: (udev-worker) Tainted: G W 6.15.0-00005-gcacccfab15bd-dirty #59 PREEMPT 
[   36.176587] Tainted: [W]=WARN
[   36.176727] Hardware name: linux,dummy-virt (DT)
[   36.176964] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   36.177301] pc : __task_rq_lock+0xf8/0x128
[   36.177576] lr : __task_rq_lock+0xf4/0x128
...
[   36.181314] Call trace:
[   36.181510]  __task_rq_lock+0xf8/0x128 (P)
[   36.181824]  sched_ttwu_pending+0x2d8/0x378
[   36.182020]  __flush_smp_call_function_queue+0x138/0x37c
[   36.182222]  generic_smp_call_function_single_interrupt+0x14/0x20
[   36.182440]  ipi_handler+0x254/0x2bc
[   36.182585]  handle_percpu_devid_irq+0xa8/0x2d4
[   36.182780]  handle_irq_desc+0x34/0x58
[   36.182942]  generic_handle_domain_irq+0x1c/0x28
[   36.183109]  gic_handle_irq+0x40/0xe0
[   36.183289]  call_on_irq_stack+0x24/0x64
[   36.183441]  do_interrupt_handler+0x80/0x84
[   36.183647]  el1_interrupt+0x34/0x70
[   36.183795]  el1h_64_irq_handler+0x18/0x24
[   36.184002]  el1h_64_irq+0x6c/0x70

[...]

kernel/sched/sched.h static inline int task_on_rq_migrating(s · line 2315

Quoted 6 lines
kernel/sched/sched.h +0 -0
@@ -2313,6 +2313,7 @@ static inline int task_on_rq_migrating(s
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
 
 #define WF_ON_CPU		0x0100

Looks like there is no specific handling for WF_ON_CPU yet?

Quoted 1 line
kernel/sched/sched.h +1 -0
+#define WF_DELAYED		0x0200

[...]

author maintainer
#20 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 22 lines · Dietmar Eggemann

On 20/05/2025 11:45, Peter Zijlstra wrote:

[...]

Quoted 4 lines
kernel/sched/core.c +0 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
 	update_rq_clock(rq);
 
 	llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {

kernel/sched/core.c void sched_ttwu_pending(void *arg) · line 3871

Quoted 10 lines
kernel/sched/core.c +10 -0
+		struct rq *p_rq = task_rq(p);
+		int ret;
+
+		/*
+		 * This is the ttwu_runnable() case. Notably it is possible for
+		 * on-rq entities to get migrated -- even sched_delayed ones.
+		 */
+		if (unlikely(p_rq != rq)) {
+			rq_unlock(rq, &rf);
+			p_rq = __task_rq_lock(p, &rf);

I always get this fairly early with TTWU_QUEUE_DELAYED enabled, related to p->pi_lock not held in wakeup from interrupt.

Quoted 1 line
[   36.175285] WARNING: CPU: 0 PID: 162 at kernel/sched/core.c:679 __task_rq_lock+0xf8/0x128

Thanks, let me go have a look.

Quoted 8 lines

kernel/sched/sched.h static inline int task_on_rq_migrating(s · line 2315

Quoted 6 lines
kernel/sched/sched.h +0 -0
@@ -2313,6 +2313,7 @@ static inline int task_on_rq_migrating(s
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
 
 #define WF_ON_CPU		0x0100

Looks like there is no specific handling for WF_ON_CPU yet?

Oh, indeed. That didn't survive the tinkering and then I forgot to clean it up here. Let me go find a broom and sweep these few bits under the carpet then :-)

author maintainer
#21 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Dietmar Eggemann

On 20/05/2025 11:45, Peter Zijlstra wrote:

[...]

Quoted 4 lines
(quoted diff) +0 -0
@@ -3830,12 +3859,41 @@ void sched_ttwu_pending(void *arg)
 	update_rq_clock(rq);
 
 	llist_for_each_entry_safe(p, t, llist, wake_entry.llist) {

Review void sched_ttwu_pending(void *arg) · line 3871

Quoted 10 lines
(quoted diff) +10 -0
+		struct rq *p_rq = task_rq(p);
+		int ret;
+
+		/*
+		 * This is the ttwu_runnable() case. Notably it is possible for
+		 * on-rq entities to get migrated -- even sched_delayed ones.
+		 */
+		if (unlikely(p_rq != rq)) {
+			rq_unlock(rq, &rf);
+			p_rq = __task_rq_lock(p, &rf);

I always get this fairly early with TTWU_QUEUE_DELAYED enabled, related to p->pi_lock not held in wakeup from interrupt.

Quoted 3 lines
Quoted 1 line
[   36.175285] WARNING: CPU: 0 PID: 162 at kernel/sched/core.c:679 __task_rq_lock+0xf8/0x128

Thanks, let me go have a look.

I'm thinking this should cure things.

kernel/sched/core.c +6 -1
@@ -677,7 +677,12 @@ struct rq *__task_rq_lock(struct task_st
 {
 	struct rq *rq;
 
-	lockdep_assert_held(&p->pi_lock);
+	/*
+	 * TASK_WAKING is used to serialize the remote end of wakeup, rather
+	 * than p->pi_lock.
+	 */
+	lockdep_assert(p->__state == TASK_WAKING ||
+		       lockdep_is_held(&p->pi_lock) != LOCK_STATE_NOT_HELD);
 
 	for (;;) {
 		rq = task_rq(p);
#22 lore raw

Re: [RFC][PATCH 5/5] sched: Add ttwu_queue support for delayed tasks

Quoted 3 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Dietmar Eggemann

On 20/05/2025 11:45, Peter Zijlstra wrote:

[...]

Quoted 15 lines
Quoted 6 lines
Quoted 4 lines

I always get this fairly early with TTWU_QUEUE_DELAYED enabled, related to p->pi_lock not held in wakeup from interrupt.

[   36.175285] WARNING: CPU: 0 PID: 162 at kernel/sched/core.c:679 __task_rq_lock+0xf8/0x128

Thanks, let me go have a look.

I'm thinking this should cure things.

kernel/sched/core.c +0 -0
@@ -677,7 +677,12 @@ struct rq *__task_rq_lock(struct task_st
 {
 	struct rq *rq;
 

kernel/sched/core.c struct rq *__task_rq_lock(struct task_st · line 688

Quoted 10 lines
kernel/sched/core.c +6 -1
-	lockdep_assert_held(&p->pi_lock);
+	/*
+	 * TASK_WAKING is used to serialize the remote end of wakeup, rather
+	 * than p->pi_lock.
+	 */
+	lockdep_assert(p->__state == TASK_WAKING ||
+		       lockdep_is_held(&p->pi_lock) != LOCK_STATE_NOT_HELD);
 
 	for (;;) {
 		rq = task_rq(p);

Yes, it does. I assume we can only end up in sched_ttwu_pending()'s 'if (unlikely(p_rq != rq))' when ttwu_queue_wakelist() is called from ttwu_runnable(), i.e. for sched_delayed tasks.

author maintainer
#23 lore raw
In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

Anybody? -- If no comments I'll just stick them in sched/core or so.

#24 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

Anybody? -- If no comments I'll just stick them in sched/core or so.

Hi Peter,

I'll get all of these run on the big turin machine, should have some numbers tomorrow.

-chris

author maintainer
#25 lore raw
Quoted 2 lines · Chris Mason

I'll get all of these run on the big turin machine, should have some numbers tomorrow.

Right... so Turin. I had a quick look through our IRC logs but I couldn't find exactly which model you had, and unfortunately AMD uses the Turin name for both Zen 5c and Zen 5 Epyc :-(

Anyway, the big and obvious difference between the whole Intel and AMD machines is the L3. So far we've been looking at SKL/SPR single L3 performance, but Turin (whichever that might be) will be having many L3. With Zen5 having 8 cores per L3 and Zen5c having 16.

Additionally, your schbench -M auto thing is doing exactly the wrong thing for them. What you want is for those message threads to be spread out across the L3s, not all stuck to the first (which is what -M auto would end up doing). And then the associated worker threads would ideally stick to their respective L3s and not scatter all over the machine.

Anyway, most of the data we shared was about single socket SKL, might be we missed some obvious things for the multi-L3 case.

I'll go poke at some of the things I've so far neglected because of the single L3 focus.

#26 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Chris Mason

I'll get all of these run on the big turin machine, should have some numbers tomorrow.

Right... so Turin. I had a quick look through our IRC logs but I couldn't find exactly which model you had, and unfortunately AMD uses the Turin name for both Zen 5c and Zen 5 Epyc :-(

Looks like the one I've been testing on is the epyc variant. But, stepping back for a bit, I bisected a few regressions between 6.9 and 6.13:

- DL server
- DELAY_{DEQUEUE,ZERO}
- PSI fun (not really me, but relevant)

I think these are all important and relevant, but it was strange that none of these patches seemed to move the needle much on the turin machines (+/- futexes), so I went back to the drawing board.

Our internal 6.9 kernel was the "fast" one, and comparing it with vanilla 6.9, it turns out we'd carried some patches that significantly improved our web workload on top of vanilla 6.9.

In other words, I've been trying to find a regression that Vernet actually fixed in 6.9 already. Bisecting pointed to:

Author: David Vernet <[email protected]>
Date:   Tue May 7 08:15:32 2024 -0700

Revert "sched/fair: Remove sysctl_sched_migration_cost condition"

This reverts commit c5b0a7eefc70150caf23e37bc9d639c68c87a097.

Comparing schedstat.py output from the fast and slow kernel..this is 6.9 vs 6.13, but I'll get a comparison tomorrow where the schedstat version actually matches.

# grep balance slow.stat.out
lb_balance_not_idle: 687
lb_imbalance_not_idle: 0
lb_balance_idle: 659054
lb_imbalance_idle: 2635
lb_balance_newly_idle: 2051682
lb_imbalance_newly_idle: 500328
sbe_balanced: 0
sbf_balanced: 0
ttwu_move_balance: 0
# grep balance fast.stat.out
lb_balance_idle: 606600
lb_imbalance_idle: 1911
lb_balance_not_idle: 680
lb_imbalance_not_idle: 0
lb_balance_newly_idle: 11697
lb_imbalance_newly_idle: 22868
sbe_balanced: 0
sbf_balanced: 0
ttwu_move_balance: 0

Reverting that commit above on vanilla 6.9 makes us fast. Disabling new idle balance completely is fast on our 6.13 kernel, but reverting that one commit doesn't change much. I'll switch back to upstream and compare newidle balance behavior.

Quoted 13 lines

Anyway, the big and obvious difference between the whole Intel and AMD machines is the L3. So far we've been looking at SKL/SPR single L3 performance, but Turin (whichever that might be) will be having many L3. With Zen5 having 8 cores per L3 and Zen5c having 16.

Additionally, your schbench -M auto thing is doing exactly the wrong thing for them. What you want is for those message threads to be spread out across the L3s, not all stuck to the first (which is what -M auto would end up doing). And then the associated worker threads would ideally stick to their respective L3s and not scatter all over the machine.

Quoted 5 lines

Anyway, most of the data we shared was about single socket SKL, might be we missed some obvious things for the multi-L3 case.

I'll go poke at some of the things I've so far neglected because of the single L3 focus.

You're 100% right about all of this, and I really do want to add some better smarts to the pinning for both numa and chiplets.

-chris

#27 lore raw
In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

Anybody? -- If no comments I'll just stick them in sched/core or so.

Hi Peter,

I've tried out your series on top of 6.15 on an Ampere Altra Mt Jade dual-socket (160-core) system, which enables SCHED_CLUSTER (2-core MC domains). Sharing preliminary test results of 50 runs per setup as, so far, the data show quite a bit of run-to-run variability - not sure how useful those will be. At this point without any deep dive, which is probably needed and hopefully will come later on.

Results for average rps (60s) sorted based on P90

CFG | min | max | stdev | 90th
----+------------+------------+------------+-----------

 1  | 704577.50  | 942665.67  | 46439.49   | 891272.09
 4  | 656313.48  | 877223.85  | 47871.43   | 837693.28
 3  | 658665.75  | 859520.32  | 50257.35   | 832174.80
 5  | 630419.62  | 842170.47  | 47267.52   | 815911.81
 2  | 647163.57  | 815392.65  | 35559.98   | 783884.00

Legend:
#1 : kernel 6.9
#2 : kernel 6.15
#3 : kernel 6.15 patched def (TTWU_QUEUE_ON_CPU + NO_TTWU_QUEUE_DEFAULT) #4 : kernel 6.15 patched + TTWU_QUEUE_ON_CPU + TTWU_QUEUE_DEFAULT #5 : kernel 6.15 patched + NO_TTWU_QUEUE_ON_CPU + NO_TTWU_QUEUE_DEFAULT

---
BR
Beata

author maintainer
#28 lore raw
Quoted 10 lines · Beata Michalska
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

Quoted 6 lines
Quoted 2 lines

Anybody? -- If no comments I'll just stick them in sched/core or so.

Hi Peter,

I've tried out your series on top of 6.15 on an Ampere Altra Mt Jade dual-socket (160-core) system, which enables SCHED_CLUSTER (2-core MC domains).

Ah, that's a radically different system than what we set out with. Good to get some feedback on that indeed.

Quoted 2 lines

Sharing preliminary test results of 50 runs per setup as, so far, the data show quite a bit of run-to-run variability - not sure how useful those will be.

Yeah, I had some of that on the Skylake system, I had to disable turbo for the numbers to become stable enough to say anything much.

Quoted 15 lines

At this point without any deep dive, which is probably needed and hopefully will come later on.

Results for average rps (60s) sorted based on P90

CFG | min | max | stdev | 90th
----+------------+------------+------------+-----------

 1  | 704577.50  | 942665.67  | 46439.49   | 891272.09
 2  | 647163.57  | 815392.65  | 35559.98   | 783884.00
 3  | 658665.75  | 859520.32  | 50257.35   | 832174.80
 4  | 656313.48  | 877223.85  | 47871.43   | 837693.28
 5  | 630419.62  | 842170.47  | 47267.52   | 815911.81
Quoted 6 lines

Legend:
#1 : kernel 6.9
#2 : kernel 6.15
#3 : kernel 6.15 patched def (TTWU_QUEUE_ON_CPU + NO_TTWU_QUEUE_DEFAULT) #4 : kernel 6.15 patched + TTWU_QUEUE_ON_CPU + TTWU_QUEUE_DEFAULT #5 : kernel 6.15 patched + NO_TTWU_QUEUE_ON_CPU + NO_TTWU_QUEUE_DEFAULT

Right, minor improvement. At least its not making it worse :-)

The new toy is TTWU_QUEUE_DELAYED, and yeah, I did notice that disabling TTWU_QUEUE_ON_CPU was a bad idea.

#29 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Chris, I'm hoping your machine will agree with these numbers; it hasn't been straight sailing in that regard.

Anybody? -- If no comments I'll just stick them in sched/core or so.

My initial numbers were quite bad, roughly 50% fewer RPS than the old 6.9 kernel on the big turin machine. I need to redo things and make sure the numbers are all valid, I'll try and do that today.

-chris

reviewer Tested-by on the v1 series
#30 lore raw
In reply to Peter Zijlstra

Hello Peter,

Quoted 4 lines · Peter Zijlstra

As can be seen, the SPR is much easier to please than the SKL for whatever reason. I'm thinking we can make TTWU_QUEUE_DELAYED default on, but I suspect TTWU_QUEUE_DEFAULT might be a harder sell -- we'd need to run more than this one benchmark.

I haven't tried toggling any of the newly added SCHED_FEAT() yet. Following are the numbers for the out of the box variant:

tl;dr Minor improvements across the board; no noticeable regressions except for a few schbench datapoints but they also have a high run-to-run variance so we should be good.

o Machine details

- 3rd Generation EPYC System
- 2 sockets each with 64C/128T
- NPS1 (Each socket is a NUMA node)
- C2 Disabled (POLL and C1(MWAIT) remained enabled)

o Kernel details

tip: tip:sched/core at commit 914873bc7df9 ("Merge tag

          'x86-build-2025-05-25' of
          git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")

ttwu_opt: tip + this series as is

o Benchmark results

    ==================================================================
    Test          : hackbench
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:           tip[pct imp](CV)       ttwu_opt[pct imp](CV)
     1-groups     1.00 [ -0.00](13.74)     0.92 [  7.68]( 6.04)
     2-groups     1.00 [ -0.00]( 9.58)     1.04 [ -3.56]( 4.96)
     4-groups     1.00 [ -0.00]( 2.10)     1.01 [ -1.30]( 2.27)
     8-groups     1.00 [ -0.00]( 1.51)     0.99 [  1.26]( 1.70)
    16-groups     1.00 [ -0.00]( 1.10)     0.97 [  3.01]( 1.62)


    ==================================================================
    Test          : tbench
    Units         : Normalized throughput
    Interpretation: Higher is better
    Statistic     : AMean
    ==================================================================
    Clients:    tip[pct imp](CV)       ttwu_opt[pct imp](CV)
        1     1.00 [  0.00]( 0.82)     1.04 [  4.33]( 1.84)
        2     1.00 [  0.00]( 1.13)     1.06 [  5.52]( 1.04)
        4     1.00 [  0.00]( 1.12)     1.05 [  5.41]( 0.53)
        8     1.00 [  0.00]( 0.93)     1.06 [  5.72]( 0.47)
       16     1.00 [  0.00]( 0.38)     1.07 [  6.99]( 0.50)
       32     1.00 [  0.00]( 0.66)     1.05 [  4.68]( 1.79)
       64     1.00 [  0.00]( 1.18)     1.06 [  5.53]( 0.37)
      128     1.00 [  0.00]( 1.12)     1.06 [  5.52]( 0.13)
      256     1.00 [  0.00]( 0.42)     0.99 [ -0.83]( 1.01)
      512     1.00 [  0.00]( 0.14)     1.01 [  1.06]( 0.13)
     1024     1.00 [  0.00]( 0.26)     1.02 [  1.82]( 0.41)


    ==================================================================
    Test          : stream-10
    Units         : Normalized Bandwidth, MB/s
    Interpretation: Higher is better
    Statistic     : HMean
    ==================================================================
    Test:       tip[pct imp](CV)       ttwu_opt[pct imp](CV)
     Copy     1.00 [  0.00]( 8.37)     0.97 [ -2.79]( 9.17)
    Scale     1.00 [  0.00]( 2.85)     1.00 [  0.12]( 2.91)
      Add     1.00 [  0.00]( 3.39)     0.98 [ -2.36]( 4.85)
    Triad     1.00 [  0.00]( 6.39)     1.01 [  1.45]( 8.42)


    ==================================================================
    Test          : stream-100
    Units         : Normalized Bandwidth, MB/s
    Interpretation: Higher is better
    Statistic     : HMean
    ==================================================================
    Test:       tip[pct imp](CV)       ttwu_opt[pct imp](CV)
     Copy     1.00 [  0.00]( 3.91)     0.98 [ -1.84]( 2.07)
    Scale     1.00 [  0.00]( 4.34)     0.96 [ -3.80]( 6.38)
      Add     1.00 [  0.00]( 4.14)     0.97 [ -3.04]( 6.31)
    Triad     1.00 [  0.00]( 1.00)     0.98 [ -2.36]( 2.60)


    ==================================================================
    Test          : netperf
    Units         : Normalized Througput
    Interpretation: Higher is better
    Statistic     : AMean
    ==================================================================
    Clients:         tip[pct imp](CV)       ttwu_opt[pct imp](CV)
     1-clients     1.00 [  0.00]( 0.41)     1.06 [  5.63]( 1.17)
     2-clients     1.00 [  0.00]( 0.58)     1.06 [  6.25]( 0.85)
     4-clients     1.00 [  0.00]( 0.35)     1.06 [  5.59]( 0.49)
     8-clients     1.00 [  0.00]( 0.48)     1.06 [  5.76]( 0.81)
    16-clients     1.00 [  0.00]( 0.66)     1.06 [  5.95]( 0.69)
    32-clients     1.00 [  0.00]( 1.15)     1.06 [  5.84]( 1.34)
    64-clients     1.00 [  0.00]( 1.38)     1.05 [  5.20]( 1.50)
    128-clients    1.00 [  0.00]( 0.87)     1.04 [  4.39]( 1.03)
    256-clients    1.00 [  0.00]( 5.36)     1.00 [  0.10]( 3.48)
    512-clients    1.00 [  0.00](54.39)     0.98 [ -1.93](52.45)


    ==================================================================
    Test          : schbench
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       ttwu_opt[pct imp](CV)
      1     1.00 [ -0.00]( 8.54)     0.89 [ 10.87](35.39)
      2     1.00 [ -0.00]( 1.15)     0.88 [ 12.00]( 4.55)
      4     1.00 [ -0.00](13.46)     0.96 [  4.17](10.60)
      8     1.00 [ -0.00]( 7.14)     0.84 [ 15.79]( 8.44)
     16     1.00 [ -0.00]( 3.49)     1.08 [ -8.47]( 4.69)
     32     1.00 [ -0.00]( 1.06)     1.10 [ -9.57]( 2.91)
     64     1.00 [ -0.00]( 5.48)     1.25 [-25.00]( 5.36)
    128     1.00 [ -0.00](10.45)     1.18 [-17.99](12.54)
    256     1.00 [ -0.00](31.14)     1.28 [-27.79](17.66)
    512     1.00 [ -0.00]( 1.52)     1.01 [ -0.51]( 2.78)


    ==================================================================
    Test          : new-schbench-requests-per-second
    Units         : Normalized Requests per second
    Interpretation: Higher is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       ttwu_opt[pct imp](CV)
      1     1.00 [  0.00]( 1.07)     1.00 [  0.29]( 0.00)
      2     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.15)
      4     1.00 [  0.00]( 0.00)     1.00 [ -0.29]( 0.15)
      8     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)
     16     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
     32     1.00 [  0.00]( 3.41)     0.99 [ -0.95]( 2.06)
     64     1.00 [  0.00]( 1.05)     0.92 [ -7.58]( 9.01)
    128     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
    256     1.00 [  0.00]( 0.72)     1.00 [ -0.31]( 0.42)
    512     1.00 [  0.00]( 0.57)     1.00 [  0.00]( 0.45)


    ==================================================================
    Test          : new-schbench-wakeup-latency
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       ttwu_opt[pct imp](CV)
      1     1.00 [ -0.00]( 9.11)     0.75 [ 25.00](11.08)
      2     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 3.78)
      4     1.00 [ -0.00]( 3.78)     0.93 [  7.14]( 3.87)
      8     1.00 [ -0.00]( 0.00)     1.08 [ -8.33](12.91)
     16     1.00 [ -0.00]( 7.56)     0.92 [  7.69](11.71)
     32     1.00 [ -0.00](15.11)     1.07 [ -6.67]( 3.30)
     64     1.00 [ -0.00]( 9.63)     1.00 [ -0.00]( 8.15)
    128     1.00 [ -0.00]( 4.86)     0.89 [ 11.06]( 7.83)
    256     1.00 [ -0.00]( 2.34)     1.00 [  0.20]( 0.10)
    512     1.00 [ -0.00]( 0.40)     1.00 [  0.38]( 0.20)


    ==================================================================
    Test          : new-schbench-request-latency
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       ttwu_opt[pct imp](CV)
      1     1.00 [ -0.00]( 2.73)     0.98 [  2.08]( 1.04)
      2     1.00 [ -0.00]( 0.87)     1.05 [ -5.40]( 3.10)
      4     1.00 [ -0.00]( 1.21)     0.99 [  0.54]( 1.27)
      8     1.00 [ -0.00]( 0.27)     0.99 [  0.79]( 2.14)
     16     1.00 [ -0.00]( 4.04)     1.01 [ -0.53]( 0.55)
     32     1.00 [ -0.00]( 7.35)     1.10 [ -9.97](21.10)
     64     1.00 [ -0.00]( 3.54)     1.03 [ -2.89]( 1.55)
    128     1.00 [ -0.00]( 0.37)     0.99 [  0.62]( 0.00)
    256     1.00 [ -0.00]( 9.57)     0.92 [  8.36]( 2.22)
    512     1.00 [ -0.00]( 1.82)     1.01 [ -1.23]( 0.94)


    ==================================================================
    Test          : Various longer running benchmarks
    Units         : %diff in throughput reported
    Interpretation: Higher is better
    Statistic     : Median
    ==================================================================
    Benchmarks:                 %diff
    ycsb-cassandra              -0.05%
    ycsb-mongodb                -0.80%

    deathstarbench-1x            2.44%
    deathstarbench-2x            5.47%
    deathstarbench-3x            0.36%
    deathstarbench-6x            1.14%

    hammerdb+mysql 16VU          1.08%
    hammerdb+mysql 64VU         -0.43%
Quoted 5 lines

Anyway, the patches are stable (finally!, I hope, knock on wood) but in a somewhat rough state. At the very least the last patch is missing ttwu_stat(), still need to figure out how to account it ;-)

Since TTWU_QUEUE_DELAYED is off by defaults, feel free to include:

if you are planning on retaining the current defaults for the SCHED_FEATs. I'll get back with numbers for TTWU_QUEUE_DELAYED and TTWU_QUEUE_DEFAULT soon.

Signature · 2 lines
Thanks and Regards,
Prateek
#31 lore raw

Hello Peter,

Quoted 9 lines · K Prateek Nayak

Hello Peter,

Quoted 4 lines · Peter Zijlstra

As can be seen, the SPR is much easier to please than the SKL for whatever reason. I'm thinking we can make TTWU_QUEUE_DELAYED default on, but I suspect TTWU_QUEUE_DEFAULT might be a harder sell -- we'd need to run more than this one benchmark.

I haven't tried toggling any of the newly added SCHED_FEAT() yet.

Here are the full results:

tldr;

- schbench (old) has a consistent regression for 16, 32, 64, 128, 256 workers (> CCX size, < Overloaded) except for with 256 workers case with TTWU_QUEUE_DEFAULT which shows an improvement.

- new schebench has few regressions around 32, 64, and 128 workers for wakeup and request latency.

- Most others benchmarks show minor improvements / regressions but nothing serious.

o Variants

"DELAYED" enables "TTWU_QUEUE_DELAYED" alone, "DEFAULT" enables "TTWU_QUEUE_DEFAULT" alone, and "BOTH" variant enables both. vanilla was shared previously which is same as out of box with no changes made to the sched features.

o Benchmark numbers

    ==================================================================
    Test          : hackbench
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:           tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
     1-groups     1.00 [ -0.00](13.74)     0.92 [  7.68]( 6.04)     0.95 [  5.12](10.12)     1.02 [ -1.92]( 6.70)     0.95 [  4.90]( 5.28)
     2-groups     1.00 [ -0.00]( 9.58)     1.04 [ -3.56]( 4.96)     1.03 [ -3.12]( 5.12)     0.98 [  1.56]( 4.30)     1.01 [ -1.11]( 5.78)
     4-groups     1.00 [ -0.00]( 2.10)     1.01 [ -1.30]( 2.27)     1.01 [ -1.09]( 2.68)     1.00 [ -0.43]( 2.58)     1.01 [ -0.65]( 1.38)
     8-groups     1.00 [ -0.00]( 1.51)     0.99 [  1.26]( 1.70)     0.99 [  0.95]( 4.92)     0.97 [  3.15]( 1.60)     1.00 [ -0.00]( 3.67)
    16-groups     1.00 [ -0.00]( 1.10)     0.97 [  3.01]( 1.62)     0.96 [  3.77]( 1.42)     0.95 [  4.60]( 0.67)     0.96 [  4.44]( 1.10)
    
    
    ==================================================================
    Test          : tbench
    Units         : Normalized throughput
    Interpretation: Higher is better
    Statistic     : AMean
    ==================================================================
    Clients:    tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
        1     1.00 [  0.00]( 0.82)     1.04 [  4.33]( 1.84)     1.06 [  5.97]( 0.42)     1.06 [  6.12]( 1.02)     1.06 [  5.54]( 0.73)
        2     1.00 [  0.00]( 1.13)     1.06 [  5.52]( 1.04)     1.07 [  7.17]( 0.42)     1.07 [  6.81]( 0.30)     1.08 [  7.96]( 0.39)
        4     1.00 [  0.00]( 1.12)     1.05 [  5.41]( 0.53)     1.07 [  7.39]( 0.67)     1.06 [  6.45]( 0.91)     1.07 [  7.36]( 0.63)
        8     1.00 [  0.00]( 0.93)     1.06 [  5.72]( 0.47)     1.07 [  6.90]( 0.24)     1.07 [  7.09]( 1.45)     1.07 [  6.94]( 0.45)
       16     1.00 [  0.00]( 0.38)     1.07 [  6.99]( 0.50)     1.05 [  4.95]( 0.98)     1.05 [  5.39]( 0.71)     1.05 [  5.43]( 1.05)
       32     1.00 [  0.00]( 0.66)     1.05 [  4.68]( 1.79)     1.06 [  5.70]( 0.54)     1.07 [  6.93]( 2.39)     1.03 [  3.17]( 1.06)
       64     1.00 [  0.00]( 1.18)     1.06 [  5.53]( 0.37)     1.04 [  4.05]( 0.84)     1.07 [  7.35]( 1.57)     1.06 [  5.62]( 1.13)
      128     1.00 [  0.00]( 1.12)     1.06 [  5.52]( 0.13)     1.05 [  4.94]( 0.75)     1.08 [  7.56]( 0.81)     1.05 [  4.80]( 0.55)
      256     1.00 [  0.00]( 0.42)     0.99 [ -0.83]( 1.01)     0.99 [ -0.58]( 0.57)     1.00 [  0.06]( 0.68)     1.00 [  0.03]( 1.47)
      512     1.00 [  0.00]( 0.14)     1.01 [  1.06]( 0.13)     1.02 [  1.67]( 0.18)     1.03 [  2.62]( 0.28)     1.02 [  2.17]( 0.33)
     1024     1.00 [  0.00]( 0.26)     1.02 [  1.82]( 0.41)     1.02 [  2.48]( 0.27)     1.03 [  3.38]( 0.37)     1.01 [  1.39]( 0.03)
    
    
    ==================================================================
    Test          : stream-10
    Units         : Normalized Bandwidth, MB/s
    Interpretation: Higher is better
    Statistic     : HMean
    ==================================================================
    Test:       tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
     Copy     1.00 [  0.00]( 8.37)     0.97 [ -2.79]( 9.17)     0.99 [ -1.29]( 4.68)     1.01 [  1.25]( 4.86)     0.99 [ -0.66]( 9.29)
    Scale     1.00 [  0.00]( 2.85)     1.00 [  0.12]( 2.91)     0.99 [ -1.34]( 5.55)     1.00 [ -0.20]( 3.38)     0.98 [ -2.09]( 5.33)
      Add     1.00 [  0.00]( 3.39)     0.98 [ -2.36]( 4.85)     0.98 [ -2.32]( 5.23)     1.00 [  0.10]( 3.17)     0.98 [ -1.99]( 4.73)
    Triad     1.00 [  0.00]( 6.39)     1.01 [  1.45]( 8.42)     1.00 [ -0.38]( 8.28)     1.05 [  4.69]( 5.66)     1.06 [  6.02]( 4.53)
    
    
    ==================================================================
    Test          : stream-100
    Units         : Normalized Bandwidth, MB/s
    Interpretation: Higher is better
    Statistic     : HMean
    ==================================================================
    Test:       tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
     Copy     1.00 [  0.00]( 3.91)     0.98 [ -1.84]( 2.07)     0.98 [ -2.06]( 6.75)     1.01 [  1.31]( 2.86)     1.02 [  2.12]( 3.30)
    Scale     1.00 [  0.00]( 4.34)     0.96 [ -3.80]( 6.38)     0.97 [ -2.88]( 6.99)     0.97 [ -2.62]( 5.70)     1.00 [ -0.37]( 3.94)
      Add     1.00 [  0.00]( 4.14)     0.97 [ -3.04]( 6.31)     0.97 [ -3.14]( 6.91)     0.99 [ -0.79]( 4.24)     1.00 [ -0.35]( 4.06)
    Triad     1.00 [  0.00]( 1.00)     0.98 [ -2.36]( 2.60)     0.96 [ -3.80]( 6.15)     0.99 [ -0.61]( 1.33)     0.97 [ -3.05]( 5.48)
    
    
    ==================================================================
    Test          : netperf
    Units         : Normalized Througput
    Interpretation: Higher is better
    Statistic     : AMean
    ==================================================================
    Clients:         tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
     1-clients     1.00 [  0.00]( 0.41)     1.06 [  5.63]( 1.17)     1.06 [  6.03]( 0.53)     1.09 [  8.63]( 0.79)     1.06 [  6.36]( 0.09)
     2-clients     1.00 [  0.00]( 0.58)     1.06 [  6.25]( 0.85)     1.05 [  5.47]( 0.83)     1.08 [  8.24]( 1.29)     1.05 [  5.15]( 0.57)
     4-clients     1.00 [  0.00]( 0.35)     1.06 [  5.59]( 0.49)     1.05 [  5.06]( 0.65)     1.08 [  8.15]( 0.82)     1.05 [  5.46]( 0.62)
     8-clients     1.00 [  0.00]( 0.48)     1.06 [  5.76]( 0.81)     1.05 [  5.26]( 0.71)     1.08 [  8.19]( 0.60)     1.05 [  5.34]( 0.80)
    16-clients     1.00 [  0.00]( 0.66)     1.06 [  5.95]( 0.69)     1.06 [  5.52]( 0.78)     1.08 [  8.31]( 0.86)     1.06 [  5.76]( 0.48)
    32-clients     1.00 [  0.00]( 1.15)     1.06 [  5.84]( 1.34)     1.06 [  5.57]( 0.96)     1.08 [  8.30]( 0.90)     1.06 [  5.66]( 1.45)
    64-clients     1.00 [  0.00]( 1.38)     1.05 [  5.20]( 1.50)     1.05 [  4.67]( 1.39)     1.07 [  7.43]( 1.47)     1.05 [  5.18]( 1.48)
    128-clients    1.00 [  0.00]( 0.87)     1.04 [  4.39]( 1.03)     1.04 [  4.43]( 0.98)     1.06 [  5.98]( 1.01)     1.05 [  4.60]( 1.06)
    256-clients    1.00 [  0.00]( 5.36)     1.00 [  0.10]( 3.48)     1.00 [  0.09]( 4.22)     1.01 [  0.71]( 3.18)     1.01 [  1.25]( 3.69)
    512-clients    1.00 [  0.00](54.39)     0.98 [ -1.93](52.45)     1.00 [ -0.35](53.30)     1.02 [  1.75](54.93)     1.02 [  1.76](55.71)
    
    
    ==================================================================
    Test          : schbench
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
      1     1.00 [ -0.00]( 8.54)     0.89 [ 10.87](35.39)     0.78 [ 21.74](34.41)     0.91 [  8.70](12.44)     0.72 [ 28.26](26.70)
      2     1.00 [ -0.00]( 1.15)     0.88 [ 12.00]( 4.55)     0.78 [ 22.00]( 6.61)     0.90 [ 10.00]( 5.75)     0.82 [ 18.00](17.98)
      4     1.00 [ -0.00](13.46)     0.96 [  4.17](10.60)     1.00 [ -0.00]( 8.54)     0.96 [  4.17]( 3.30)     0.98 [  2.08]( 8.19)
      8     1.00 [ -0.00]( 7.14)     0.84 [ 15.79]( 8.44)     0.98 [  1.75]( 3.67)     0.95 [  5.26]( 4.99)     0.91 [  8.77]( 2.92)
     16     1.00 [ -0.00]( 3.49)     1.08 [ -8.47]( 4.69)     1.07 [ -6.78]( 0.92)     1.07 [ -6.78]( 0.91)     1.07 [ -6.78]( 3.27)
     32     1.00 [ -0.00]( 1.06)     1.10 [ -9.57]( 2.91)     1.07 [ -7.45]( 2.97)     1.07 [ -7.45]( 4.23)     1.05 [ -5.32]( 7.80)
     64     1.00 [ -0.00]( 5.48)     1.25 [-25.00]( 5.36)     1.17 [-17.44]( 1.44)     1.23 [-23.26]( 2.79)     1.20 [-19.77]( 2.19)
    128     1.00 [ -0.00](10.45)     1.18 [-17.99](12.54)     1.16 [-16.36](21.21)     1.13 [-12.85](12.71)     1.09 [ -8.64]( 3.05)
    256     1.00 [ -0.00](31.14)     1.28 [-27.79](17.66)     0.84 [ 16.21](32.14)     1.19 [-19.21]( 1.68)     1.07 [ -6.86]( 7.48)
    512     1.00 [ -0.00]( 1.52)     1.01 [ -0.51]( 2.78)     0.97 [  3.03]( 2.91)     0.98 [  1.77]( 1.07)     1.01 [ -0.51]( 1.01)
    
    
    ==================================================================
    Test          : new-schbench-requests-per-second
    Units         : Normalized Requests per second
    Interpretation: Higher is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
      1     1.00 [  0.00]( 1.07)     1.00 [  0.29]( 0.00)     1.00 [  0.29]( 0.15)     0.99 [ -0.59]( 0.46)     1.00 [  0.29]( 0.30)
      2     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)
      4     1.00 [  0.00]( 0.00)     1.00 [ -0.29]( 0.15)     1.00 [  0.00]( 0.00)     1.00 [ -0.29]( 0.15)     1.00 [  0.00]( 0.00)
      8     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.15)     1.00 [  0.29]( 0.00)     1.00 [  0.00]( 0.40)     1.00 [  0.29]( 0.15)
     16     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.15)     1.00 [  0.00]( 0.00)
     32     1.00 [  0.00]( 3.41)     0.99 [ -0.95]( 2.06)     0.98 [ -2.23]( 3.41)     0.98 [ -2.23]( 3.31)     1.03 [  2.54]( 0.32)
     64     1.00 [  0.00]( 1.05)     0.92 [ -7.58]( 9.01)     0.86 [-13.92](11.30)     1.00 [  0.00]( 4.74)     1.00 [ -0.38]( 9.98)
    128     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.00]( 0.00)     1.00 [  0.38]( 0.00)
    256     1.00 [  0.00]( 0.72)     1.00 [ -0.31]( 0.42)     1.01 [  1.23]( 1.33)     1.01 [  0.61]( 0.83)     1.01 [  0.92]( 1.36)
    512     1.00 [  0.00]( 0.57)     1.00 [  0.00]( 0.45)     0.99 [ -0.72]( 1.18)     1.00 [  0.48]( 0.33)     1.01 [  1.44]( 0.49)
    
    
    ==================================================================
    Test          : new-schbench-wakeup-latency
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
      1     1.00 [ -0.00]( 9.11)     0.75 [ 25.00](11.08)     0.69 [ 31.25]( 8.13)     0.75 [ 25.00](11.08)     0.62 [ 37.50]( 8.94)
      2     1.00 [ -0.00]( 0.00)     1.00 [ -0.00]( 3.78)     0.86 [ 14.29]( 7.45)     0.93 [  7.14]( 3.87)     0.79 [ 21.43]( 4.84)
      4     1.00 [ -0.00]( 3.78)     0.93 [  7.14]( 3.87)     0.79 [ 21.43]( 4.56)     0.93 [  7.14]( 0.00)     0.79 [ 21.43]( 8.85)
      8     1.00 [ -0.00]( 0.00)     1.08 [ -8.33](12.91)     0.92 [  8.33]( 0.00)     0.83 [ 16.67](18.23)     1.08 [ -8.33](12.91)
     16     1.00 [ -0.00]( 7.56)     0.92 [  7.69](11.71)     0.85 [ 15.38](12.06)     1.08 [ -7.69](11.92)     0.85 [ 15.38](12.91)
     32     1.00 [ -0.00](15.11)     1.07 [ -6.67]( 3.30)     1.00 [ -0.00](19.06)     1.00 [ -0.00](15.11)     0.80 [ 20.00]( 4.43)
     64     1.00 [ -0.00]( 9.63)     1.00 [ -0.00]( 8.15)     1.00 [ -0.00]( 5.34)     1.05 [ -5.00]( 7.75)     0.90 [ 10.00]( 9.94)
    128     1.00 [ -0.00]( 4.86)     0.89 [ 11.06]( 7.83)     0.91 [  8.54]( 7.87)     0.88 [ 12.06]( 8.73)     0.86 [ 14.07]( 5.01)
    256     1.00 [ -0.00]( 2.34)     1.00 [  0.20]( 0.10)     1.04 [ -4.50]( 4.59)     1.03 [ -2.90]( 1.95)     1.04 [ -3.70]( 4.13)
    512     1.00 [ -0.00]( 0.40)     1.00 [  0.38]( 0.20)     1.00 [  0.38]( 0.20)     0.99 [  0.77]( 0.20)     1.00 [ -0.00]( 0.40)
    
    
    ==================================================================
    Test          : new-schbench-request-latency
    Units         : Normalized 99th percentile latency in us
    Interpretation: Lower is better
    Statistic     : Median
    ==================================================================
    #workers: tip[pct imp](CV)       vanilla[pct imp](CV)     DELAYED[pct imp](CV)     DEFUALT[pct imp](CV)      BOTH[pct imp](CV)
      1     1.00 [ -0.00]( 2.73)     0.98 [  2.08]( 1.04)     0.99 [  1.30]( 1.07)     1.02 [ -1.82]( 0.00)     1.01 [ -1.30]( 3.10)
      2     1.00 [ -0.00]( 0.87)     1.05 [ -5.40]( 3.10)     1.02 [ -1.89]( 1.58)     1.01 [ -1.08]( 2.76)     1.02 [ -1.62]( 1.45)
      4     1.00 [ -0.00]( 1.21)     0.99 [  0.54]( 1.27)     0.99 [  1.08]( 1.67)     1.01 [ -1.21]( 1.21)     1.01 [ -1.35]( 1.91)
      8     1.00 [ -0.00]( 0.27)     0.99 [  0.79]( 2.14)     0.98 [  2.37]( 0.72)     0.99 [  1.05]( 2.53)     0.99 [  0.79]( 1.12)
     16     1.00 [ -0.00]( 4.04)     1.01 [ -0.53]( 0.55)     1.01 [ -0.80]( 1.08)     1.00 [ -0.27]( 0.36)     0.99 [  0.53]( 0.50)
     32     1.00 [ -0.00]( 7.35)     1.10 [ -9.97](21.10)     1.01 [ -0.66](10.27)     1.25 [-25.36](21.41)     0.90 [  9.52]( 2.08)
     64     1.00 [ -0.00]( 3.54)     1.03 [ -2.89]( 1.55)     1.02 [ -2.00]( 0.98)     1.01 [ -0.67]( 3.62)     1.01 [ -0.89]( 4.98)
    128     1.00 [ -0.00]( 0.37)     0.99 [  0.62]( 0.00)     0.99 [  0.72]( 0.11)     0.99 [  0.62]( 0.11)     0.99 [  0.83]( 0.11)
    256     1.00 [ -0.00]( 9.57)     0.92 [  8.36]( 2.22)     1.03 [ -3.11](12.58)     1.05 [ -5.02]( 8.36)     1.00 [ -0.00](11.71)
    512     1.00 [ -0.00]( 1.82)     1.01 [ -1.23]( 0.94)     1.02 [ -2.45]( 1.53)     1.00 [  0.35]( 0.83)     1.02 [ -1.93]( 1.40)
    
    ==================================================================
    Test          : Various longer running benchmarks
    Units         : %diff in throughput reported
    Interpretation: Higher is better
    Statistic     : Median
    ==================================================================
    Benchmarks:                 vanilla     DELAYED   DEFAULT    BOTH
    ycsb-cassandra              -0.05%       0.65%    -0.49%    -0.48%
    ycsb-mongodb                -0.80%      -0.85%    -1.00%    -0.98%
     
    deathstarbench-1x            2.44%       1.54%     1.65%     0.18%
    deathstarbench-2x            5.47%       4.88%     7.92%     6.75%
    deathstarbench-3x            0.36%       1.74%    -1.75%     0.31%
    deathstarbench-6x            1.14%       1.94%     2.24%     1.58%
    
    hammerdb+mysql 16VU          1.08%       5.21%     2.69%     3.80%
    hammerdb+mysql 64VU         -0.43%      -0.31%     2.12%    -0.25%
Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#32 lore raw
Quoted 7 lines · K Prateek Nayak

- schbench (old) has a consistent regression for 16, 32, 64, 128, 256 workers (> CCX size, < Overloaded) except for with 256 workers case with TTWU_QUEUE_DEFAULT which shows an improvement.

- new schebench has few regressions around 32, 64, and 128 workers for wakeup and request latency.

Right, so I actually made Chris' favourite workloads worse with these patches :/

Let me go try this again..