timers/migration: Prioritize lower capacity CPUs as migrators

Hi,

It has been reported that the timer migration tree wasn't too helpful on asymetric systems, with timers migrating too often to big capacity CPUs, waking them more frequently from deep idle state.

Splitting the tree into per CPU capacity hierarchies doesn't seem to improve the situation.

So here is another proposal: gather CPUs belonging to the same capacity under the same groups and finally join those capacity groups under a common root. Exactly in the same way it is done for NUMA systems.

And then always pick up the lowest capacity non-idle CPU group as the migrator with the hope that handling timers will keep lower capacity CPUs busy enough to increase their chances to become global idle migrators.

Please give it a try but make sure you have fewer than 9 capacities on your system. Handling more will require a bit more throughts. Let's see first if this is the right way to go.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/migration-capacity

HEAD: 28f02a6747ac4343f3596aaa3b749dea95577478
Thanks,
Frederic
---

Frederic Weisbecker (6):
timers/migration: Revert per CPU capacity hierarchy timers/migration: Defer initialization after capacity topology is setup sched/topology: Account asym capacities number timers/migration: Group CPUs per capacity
timers/migration: Prefer lower capacity groups as migrators scripts/timer_migration_tree.py: Dump mask of each group

 include/linux/sched/topology.h         |   3 +
 include/trace/events/timer_migration.h |  48 ++--
 kernel/sched/topology.c                |  44 ++++
 kernel/time/timer_migration.c          | 393 +++++++++++++++------------------
 kernel/time/timer_migration.h          |  33 +--
 scripts/timer_migration_tree.py        |  65 +++---
 6 files changed, 288 insertions(+), 298 deletions(-)

The per CPU-capacity hierarchies may not be the right answer to solve timers migrating too often to high capacity CPUs. Revert that in favour of later code that will gather groups per capacity under a common root and prefer lower capacity groups as migrators.

---
 include/trace/events/timer_migration.h |  24 +--
 kernel/time/timer_migration.c          | 241 ++++++-------------------
 kernel/time/timer_migration.h          |  18 --
 scripts/timer_migration_tree.py        |  50 ++---
 4 files changed, 87 insertions(+), 246 deletions(-)

index 0b135e9301b1..61171b13c687 100644
@@ -33,16 +33,15 @@ TRACE_EVENT(tmigr_group_set,
 
 TRACE_EVENT(tmigr_connect_child_parent,
 
-	TP_PROTO(struct tmigr_hierarchy *hier, struct tmigr_group *child),
+	TP_PROTO(struct tmigr_group *child),
 
-	TP_ARGS(hier, child),
+	TP_ARGS(child),
 
 	TP_STRUCT__entry(
 		__field( void *,	child		)
 		__field( void *,	parent		)
 		__field( unsigned int,	lvl		)
 		__field( unsigned int,	numa_node	)
-		__field( unsigned int,	capacity	)
 		__field( unsigned int,	num_children	)
 		__field( u32,		groupmask	)
 	),

@@ -52,28 +51,26 @@ TRACE_EVENT(tmigr_connect_child_parent,
 		__entry->parent		= child->parent;
 		__entry->lvl		= child->parent->level;
 		__entry->numa_node	= child->parent->numa_node;
-		__entry->capacity	= hier->capacity;
 		__entry->num_children	= child->parent->num_children;
 		__entry->groupmask	= child->groupmask;
 	),
 
-	TP_printk("group=%p groupmask=%0x parent=%p lvl=%d numa=%d capacity=%d num_children=%d",
-		  __entry->child,  __entry->groupmask, __entry->parent, __entry->lvl,
-		  __entry->numa_node, __entry->capacity, __entry->num_children)
+	TP_printk("group=%p groupmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+		  __entry->child,  __entry->groupmask, __entry->parent,
+		  __entry->lvl, __entry->numa_node, __entry->num_children)
 );
 
 TRACE_EVENT(tmigr_connect_cpu_parent,
 
-	TP_PROTO(struct tmigr_hierarchy *hier, struct tmigr_cpu *tmc),
+	TP_PROTO(struct tmigr_cpu *tmc),
 
-	TP_ARGS(hier, tmc),
+	TP_ARGS(tmc),
 
 	TP_STRUCT__entry(
 		__field( void *,	parent		)
 		__field( unsigned int,	cpu		)
 		__field( unsigned int,	lvl		)
 		__field( unsigned int,	numa_node	)
-		__field( unsigned int,	capacity	)
 		__field( unsigned int,	num_children	)
 		__field( u32,		groupmask	)
 	),

@@ -83,14 +80,13 @@ TRACE_EVENT(tmigr_connect_cpu_parent,
 		__entry->cpu		= tmc->cpuevt.cpu;
 		__entry->lvl		= tmc->tmgroup->level;
 		__entry->numa_node	= tmc->tmgroup->numa_node;
-		__entry->capacity	= hier->capacity;
 		__entry->num_children	= tmc->tmgroup->num_children;
 		__entry->groupmask	= tmc->groupmask;
 	),
 
-	TP_printk("cpu=%d groupmask=%0x parent=%p lvl=%d numa=%d capacity=%d num_children=%d",
-		  __entry->cpu,	 __entry->groupmask, __entry->parent, __entry->lvl,
-		  __entry->numa_node, __entry->capacity, __entry->num_children)
+	TP_printk("cpu=%d groupmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+		  __entry->cpu,	 __entry->groupmask, __entry->parent,
+		  __entry->lvl, __entry->numa_node, __entry->num_children)
 );
 
 DECLARE_EVENT_CLASS(tmigr_group_and_cpu,

index 31735dd52327..4c0073f3d321 100644
@@ -5,24 +5,6 @@
 /* Per group capacity. Must be a power of 2! */
 #define TMIGR_CHILDREN_PER_GROUP 8
 
-/**
- * struct tmigr_hierarchy - a hierarchy associated to a given CPU capacity.
- *                          Homogeneous systems have only one hierarchy.
- *                          Heterogenous have one hierarchy per CPU capacity.
- * @cpumask:	CPUs belonging to this hierarchy
- * @root:	The current root of the hierarchy
- * @capacity:	CPU capacity associated to this hierarchy
- * @node:	Node in the global hierarchy list
- * @level_list:	Per level lists of tmigr groups
- */
-struct tmigr_hierarchy {
-	struct cpumask		*cpumask;
-	struct tmigr_group	*root;
-	unsigned long		capacity;
-	struct list_head	node;
-	struct list_head	level_list[];
-};
-
 /**
  * struct tmigr_event - a timer event associated to a CPU
  * @nextevt:	The node to enqueue an event in the parent group queue

2.53.0

In order for the timer migration code to get informations from the CPU capacity list, the tree must be prepared after the scheduler topology is initialized.

Defer this part and the hotplug callbacks to a late initcall. Two new cases are taken care of now that the registration of the callbacks is made after CPUs have booted:

* The prepare callback may now run on the target.

* A subsequent root level can be created even if no CPU is available in the hierarchy because prepare callbacks are called on all CPUs before online callbacks.

---
 kernel/time/timer_migration.c | 92 +++++++++++++++++++++++++----------
 1 file changed, 67 insertions(+), 25 deletions(-)

index 52c15affdbff..4209e695ec7b 100644
@@ -1631,7 +1631,6 @@ static int __init tmigr_init_isolation(void)
 	/* Protect against RCU torture hotplug testing */
 	return tmigr_isolated_exclude_cpumask(cpumask);
 }
-late_initcall(tmigr_init_isolation);
 
 static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl,
 			     int node)

@@ -1703,7 +1702,7 @@ static struct tmigr_group *tmigr_get_group(int node, unsigned int lvl)
 	return group;
 }
 
-static bool tmigr_init_root(struct tmigr_group *group, bool activate)
+static bool tmigr_init_root(struct tmigr_group *group, bool root_up)
 {
 	if (!group->parent && group != tmigr_root) {
 		/*

@@ -1712,7 +1711,7 @@ static bool tmigr_init_root(struct tmigr_group *group, bool activate)
 		 * created in the future and made visible before this groupmask.
 		 */
 		group->groupmask = BIT(0);
-		WARN_ON_ONCE(activate);
+		WARN_ON_ONCE(root_up);
 
 		return true;
 	}

@@ -1723,9 +1722,9 @@ static bool tmigr_init_root(struct tmigr_group *group, bool activate)
 
 static void tmigr_connect_child_parent(struct tmigr_group *child,
 				       struct tmigr_group *parent,
-				       bool activate)
+				       bool root_up)
 {
-	if (tmigr_init_root(parent, activate)) {
+	if (tmigr_init_root(parent, root_up)) {
 		/*
 		 * The previous top level had prepared its groupmask already,
 		 * simply account it in advance as the first child. If some groups

@@ -1736,7 +1735,7 @@ static void tmigr_connect_child_parent(struct tmigr_group *child,
 	}
 
 	/* Connecting old root to new root ? */
-	if (!parent->parent && activate) {
+	if (!parent->parent && root_up) {
 		/*
 		 * @child is the old top, or in case of node mismatch, some
 		 * intermediate group between the old top and the new one in

@@ -1831,7 +1830,7 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node,
 			tmc->tmgroup = group;
 			tmc->groupmask = BIT(group->num_children++);
 
-			tmigr_init_root(group, activate);
+			tmigr_init_root(group, start);
 
 			trace_tmigr_connect_cpu_parent(tmc);
 

@@ -1839,7 +1838,7 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node,
 			continue;
 		} else {
 			child = stack[i - 1];
-			tmigr_connect_child_parent(child, group, activate);
+			tmigr_connect_child_parent(child, group, start);
 		}
 	}
 

@@ -1894,8 +1893,14 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node,
 		if (state.active) {
 			data.childmask = start->groupmask;
 			__walk_groups_from(tmigr_active_up, &data, start, start->parent);
+		} else if (start) {
+			union tmigr_state state;
+
+			/* No available CPU so the old root should be inactive */
+			state.state = atomic_read(&start->migr_state);
+			WARN_ON_ONCE(state.active);
 		}
-	}
+        }
 
 	/* Root update */
 	if (list_is_singular(&tmigr_level_list[top])) {

@@ -1914,18 +1919,9 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node,
 	return err;
 }
 
-static int tmigr_add_cpu(unsigned int cpu)
+static int tmigr_connect_old_root(int cpu, struct tmigr_group *old_root, bool activate)
 {
-	struct tmigr_group *old_root = tmigr_root;
-	int node = cpu_to_node(cpu);
-	int ret;
-
-	guard(mutex)(&tmigr_mutex);
-
-	ret = tmigr_setup_groups(cpu, node, NULL, false);
-
-	/* Root has changed? Connect the old one to the new */
-	if (ret >= 0 && old_root && old_root != tmigr_root) {
+	if (activate) {
 		/*
 		 * The target CPU must never do the prepare work, except
 		 * on early boot when the boot CPU is the target. Otherwise

@@ -1933,13 +1929,55 @@ static int tmigr_add_cpu(unsigned int cpu)
 		 * the new one (nevertheless whether old top level group is
 		 * active or not) and/or release an uninitialized childmask.
 		 */
-		WARN_ON_ONCE(cpu == raw_smp_processor_id());
+		WARN_ON_ONCE(cpu == smp_processor_id());
 		/*
-		 * The (likely) current CPU is expected to be online in the hierarchy,
+		 * The current CPU is expected to be online in the hierarchy,
 		 * otherwise the old root may not be active as expected.
 		 */
-		WARN_ON_ONCE(!per_cpu_ptr(&tmigr_cpu, raw_smp_processor_id())->available);
-		ret = tmigr_setup_groups(-1, old_root->numa_node, old_root, true);
+		WARN_ON_ONCE(!__this_cpu_read(tmigr_cpu.available));
+	}
+
+	return tmigr_setup_groups(-1, old_root->numa_node, old_root, activate);
+}
+
+static long connect_old_root_work(void *arg)
+{
+	struct tmigr_group *old_root = arg;
+
+	return tmigr_connect_old_root(smp_processor_id(), old_root, true);
+}
+
+static int tmigr_add_cpu(unsigned int cpu)
+{
+	struct tmigr_group *old_root = tmigr_root;
+	int node = cpu_to_node(cpu);
+	int ret;
+
+	guard(mutex)(&tmigr_mutex);
+
+	ret = tmigr_setup_groups(cpu, node, NULL, false);
+
+	if (ret < 0 || !old_root || old_root == tmigr_root)
+		return ret;
+
+	/* Root has changed. Connect the old one to the new */
+	guard(migrate)();
+	if (cpumask_test_cpu(smp_processor_id(), tmigr_available_cpumask)) {
+		/*
+		 * If the CPU is available in the hierarchy, the old root is expected
+		 * to be active. Link and propagate to the new root.
+		 */
+		ret = tmigr_connect_old_root(cpu, old_root, true);
+	} else {
+		int target = cpumask_first(tmigr_available_cpumask);
+
+		if (target < nr_cpu_ids) {
+			/* Defer the connection to an available CPU to propagate activation */
+			ret = work_on_cpu(target, connect_old_root_work, old_root);
+		} else {
+			/* No CPU available yet, connect but don't activate */
+			ret = tmigr_connect_old_root(cpu, old_root, false);
+		}
 	}
 
 	return ret;

@@ -2044,10 +2082,14 @@ static int __init tmigr_init(void)
 	if (ret)
 		goto err;
 
+	ret = tmigr_init_isolation();
+	if (ret)
+		goto err;
+
 	return 0;
 
 err:
 	pr_err("Timer migration setup failed\n");
 	return ret;
 }
-early_initcall(tmigr_init);
+late_initcall(tmigr_init);

2.53.0

The timer migration code will need to know the number of capacities in the system and the maximum number of CPUs a capacity can contain in order to build the tree accordingly.

Prepare the relevant APIs.

---
 include/linux/sched/topology.h |  2 ++
 kernel/sched/topology.c        | 24 ++++++++++++++++++++++++
 2 files changed, 26 insertions(+)

index b5d9d7c2b8ad..88632825136e 100644
@@ -50,6 +50,8 @@ extern const struct cpumask *tl_mc_mask(struct sched_domain_topology_level *tl,
 extern const struct cpumask *tl_pkg_mask(struct sched_domain_topology_level *tl, int cpu);
 
 extern int arch_asym_cpu_priority(int cpu);
+extern int sched_asym_count(void);
+extern int sched_asym_max_cpus(void);
 
 struct sched_domain_attr {
 	int relax_domain_level;

index 622e2e01974c..3b3bd32aea40 100644
@@ -1746,6 +1746,20 @@ static inline void asym_cpu_capacity_update_data(int cpu)
 	__cpumask_set_cpu(cpu, cpu_capacity_span(entry));
 }
 
+static int asym_capacity_nr;
+
+int sched_asym_count(void)
+{
+	return asym_capacity_nr;
+}
+
+static int asym_capacity_max_cpus;
+
+int sched_asym_max_cpus(void)
+{
+	return asym_capacity_max_cpus;
+}
+
 /*
  * Build-up/update list of CPUs grouped by their capacities
  * An update requires explicit request to rebuild sched domains

@@ -1756,6 +1770,9 @@ static void asym_cpu_capacity_scan(void)
 	struct asym_cap_data *entry, *next;
 	int cpu;
 
+	asym_capacity_nr = 0;
+	asym_capacity_max_cpus = 0;
+
 	list_for_each_entry(entry, &asym_cap_list, link)
 		cpumask_clear(cpu_capacity_span(entry));
 

@@ -1766,6 +1783,13 @@ static void asym_cpu_capacity_scan(void)
 		if (cpumask_empty(cpu_capacity_span(entry))) {
 			list_del_rcu(&entry->link);
 			call_rcu(&entry->rcu, free_asym_cap_entry);
+		} else {
+			int weight;
+
+			asym_capacity_nr++;
+			weight = cpumask_weight(cpu_capacity_span(entry));
+			if (weight > asym_capacity_max_cpus)
+				asym_capacity_max_cpus = weight;
 		}
 	}
 

2.53.0

In the same way CPUs are grouped by node, group CPUs by capacity (this assumes that asymetric capacity systems aren't NUMA but reality either is or will be certainly disappointing about this).

This way, timers have fewer chances to migrate from low to high capacity CPUs, and related ground work is more likely to execute while consuming less energy.

Also in order to stimulate migration from high to low capacity CPUs, a further change will encourage low capacity global migrators.

---
 include/trace/events/timer_migration.h |  32 +++----
 kernel/time/timer_migration.c          | 125 ++++++++++++++++---------
 kernel/time/timer_migration.h          |  13 +--
 3 files changed, 103 insertions(+), 67 deletions(-)

index 61171b13c687..5484056b0012 100644
@@ -18,17 +18,17 @@ TRACE_EVENT(tmigr_group_set,
 	TP_STRUCT__entry(
 		__field( void *,	group		)
 		__field( unsigned int,	lvl		)
-		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	family	)
 	),
 
 	TP_fast_assign(
 		__entry->group		= group;
 		__entry->lvl		= group->level;
-		__entry->numa_node	= group->numa_node;
+		__entry->family		= group->family;
 	),
 
-	TP_printk("group=%p lvl=%d numa=%d",
-		  __entry->group, __entry->lvl, __entry->numa_node)
+	TP_printk("group=%p lvl=%d family=%d",
+		  __entry->group, __entry->lvl, __entry->family)
 );
 
 TRACE_EVENT(tmigr_connect_child_parent,

@@ -41,7 +41,7 @@ TRACE_EVENT(tmigr_connect_child_parent,
 		__field( void *,	child		)
 		__field( void *,	parent		)
 		__field( unsigned int,	lvl		)
-		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	family	)
 		__field( unsigned int,	num_children	)
 		__field( u32,		groupmask	)
 	),

@@ -50,14 +50,14 @@ TRACE_EVENT(tmigr_connect_child_parent,
 		__entry->child		= child;
 		__entry->parent		= child->parent;
 		__entry->lvl		= child->parent->level;
-		__entry->numa_node	= child->parent->numa_node;
+		__entry->family		= child->parent->family;
 		__entry->num_children	= child->parent->num_children;
 		__entry->groupmask	= child->groupmask;
 	),
 
-	TP_printk("group=%p groupmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+	TP_printk("group=%p groupmask=%0x parent=%p lvl=%d family=%d num_children=%d",
 		  __entry->child,  __entry->groupmask, __entry->parent,
-		  __entry->lvl, __entry->numa_node, __entry->num_children)
+		  __entry->lvl, __entry->family, __entry->num_children)
 );
 
 TRACE_EVENT(tmigr_connect_cpu_parent,

@@ -70,7 +70,7 @@ TRACE_EVENT(tmigr_connect_cpu_parent,
 		__field( void *,	parent		)
 		__field( unsigned int,	cpu		)
 		__field( unsigned int,	lvl		)
-		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	family	)
 		__field( unsigned int,	num_children	)
 		__field( u32,		groupmask	)
 	),

@@ -79,14 +79,14 @@ TRACE_EVENT(tmigr_connect_cpu_parent,
 		__entry->parent		= tmc->tmgroup;
 		__entry->cpu		= tmc->cpuevt.cpu;
 		__entry->lvl		= tmc->tmgroup->level;
-		__entry->numa_node	= tmc->tmgroup->numa_node;
+		__entry->family		= tmc->tmgroup->family;
 		__entry->num_children	= tmc->tmgroup->num_children;
 		__entry->groupmask	= tmc->groupmask;
 	),
 
-	TP_printk("cpu=%d groupmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+	TP_printk("cpu=%d groupmask=%0x parent=%p lvl=%d family=%d num_children=%d",
 		  __entry->cpu,	 __entry->groupmask, __entry->parent,
-		  __entry->lvl, __entry->numa_node, __entry->num_children)
+		  __entry->lvl, __entry->family, __entry->num_children)
 );
 
 DECLARE_EVENT_CLASS(tmigr_group_and_cpu,

@@ -99,7 +99,7 @@ DECLARE_EVENT_CLASS(tmigr_group_and_cpu,
 		__field( void *,	group		)
 		__field( void *,	parent		)
 		__field( unsigned int,	lvl		)
-		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	family	)
 		__field( u32,		childmask	)
 		__field( u8,		active		)
 		__field( u8,		migrator	)

@@ -109,15 +109,15 @@ DECLARE_EVENT_CLASS(tmigr_group_and_cpu,
 		__entry->group		= group;
 		__entry->parent		= group->parent;
 		__entry->lvl		= group->level;
-		__entry->numa_node	= group->numa_node;
+		__entry->family		= group->family;
 		__entry->childmask	= childmask;
 		__entry->active		= state.active;
 		__entry->migrator	= state.migrator;
 	),
 
-	TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x "
+	TP_printk("group=%p lvl=%d family=%d active=%0x migrator=%0x "
 		  "parent=%p childmask=%0x",
-		  __entry->group, __entry->lvl, __entry->numa_node,
+		  __entry->group, __entry->lvl, __entry->family,
 		  __entry->active, __entry->migrator,
 		  __entry->parent, __entry->childmask)
 );

index 4c0073f3d321..3f6c7a110e3c 100644
@@ -43,11 +43,12 @@ struct tmigr_event {
  * @events:		Timer queue for child events queued in the group
  * @migr_state:		State of the group (see union tmigr_state)
  * @level:		Hierarchy level of the group; Required during setup
- * @numa_node:		Required for setup only to make sure CPU and low level
- *			group information is NUMA local. It is set to NUMA node
- *			as long as the group level is per NUMA node (level <
- *			tmigr_crossnode_level); otherwise it is set to
- *			NUMA_NO_NODE
+ * @family:		Either NUMA node id or asym CPU capacity. Required for
+ *			setup only to make sure CPU and low level
+ *			group information is NUMA/capacity local. It is set to
+ *			NUMA node as long as the group level is per NUMA node
+ * 			(level < tmigr_family_level); otherwise it is set to
+ * 			the CPUs capacities.
  * @num_children:	Counter of group children to make sure the group is only
  *			filled with TMIGR_CHILDREN_PER_GROUP; Required for setup
  *			only

@@ -66,7 +67,7 @@ struct tmigr_group {
 	struct timerqueue_head	events;
 	atomic_t		migr_state;
 	unsigned int		level;
-	int			numa_node;
+	int			family;
 	unsigned int		num_children;
 	u8			groupmask;
 	struct list_head	list;

2.53.0

Pulling timers to low capacity CPUs may improve power consumption by executing non performance critical ground work there and increasing the chances to turn the small CPUs into idle global migrators if such work keeps them alive for long enough. This way low power CPUs may be woken up from deep idle instead of high power CPUs.

To implement this, migrators going idle will select the lowest alive capacity groups as new migrators. And CPUs exiting idle will force select themselves as the new migrator if their capacity is lower than the current migrator.

---
 include/linux/sched/topology.h  |  1 +
 kernel/sched/topology.c         | 20 ++++++++++++++++++++
 kernel/time/timer_migration.c   | 33 +++++++++++++++++++++++++++------
 kernel/time/timer_migration.h   |  2 ++
 scripts/timer_migration_tree.py | 24 ++++++++++++------------
 5 files changed, 62 insertions(+), 18 deletions(-)

index 88632825136e..de584bde7dee 100644
@@ -52,6 +52,7 @@ extern const struct cpumask *tl_pkg_mask(struct sched_domain_topology_level *tl,
 extern int arch_asym_cpu_priority(int cpu);
 extern int sched_asym_count(void);
 extern int sched_asym_max_cpus(void);
+extern int sched_asym_capacity_rank(unsigned long capacity);
 
 struct sched_domain_attr {
 	int relax_domain_level;

index 3b3bd32aea40..7673dfd579db 100644
@@ -1760,6 +1760,26 @@ int sched_asym_max_cpus(void)
 	return asym_capacity_max_cpus;
 }
 
+int sched_asym_capacity_rank(unsigned long capacity)
+{
+	struct asym_cap_data *entry;
+	int i = 0;
+
+	/*
+	 * Search if capacity already exits. If not, track which the entry
+	 * where we should insert to keep the list ordered descending.
+	 */
+	list_for_each_entry_reverse(entry, &asym_cap_list, link) {
+		if (capacity == entry->capacity)
+			return i;
+		i++;
+	}
+
+	WARN_ONCE(1, "Capacity %lu not found in capacity list", capacity);
+
+	return 0;
+}
+
 /*
  * Build-up/update list of CPUs grouped by their capacities
  * An update requires explicit request to rebuild sched domains

index 2c2925046f43..a16d265df33e 100644
@@ -677,11 +677,14 @@ static bool tmigr_active_up(struct tmigr_group *group,
 		newstate = curstate;
 		walk_done = true;
 
-		if (newstate.migrator == TMIGR_NONE) {
+		if (curstate.migrator == TMIGR_NONE ||
+		    (group->want_low_migrator && childmask < curstate.migrator)) {
 			newstate.migrator = childmask;
 
-			/* Changes need to be propagated */
-			walk_done = false;
+			if (curstate.migrator == TMIGR_NONE) {
+				/* Changes need to be propagated */
+				walk_done = false;
+			}
 		}
 
 		newstate.active |= childmask;

@@ -1644,6 +1647,12 @@ static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl,
 
 	group->num_children = 0;
 
+	/* Always prefer a migrator with lower capacity */
+	if (sched_asym_count() > 1 && lvl == tmigr_crossfamily_level)
+		group->want_low_migrator = true;
+	else
+		group->want_low_migrator = false;
+
 	s.migrator = TMIGR_NONE;
 	s.active = 0;
 	s.seq = 0;

@@ -1708,6 +1717,18 @@ static struct tmigr_group *tmigr_get_group(int family, unsigned int lvl)
 	return group;
 }
 
+static void tmigr_init_groupmask(struct tmigr_group *group, u8 groupmask)
+{
+	/*
+	 * Overwrite the groupmask if this is a whole capacity group so that
+	 * candidate migrators are sorted by capacity.
+	 */
+	if (sched_asym_count() > 1 && group->level == tmigr_crossfamily_level - 1)
+		groupmask = BIT(sched_asym_capacity_rank(group->family));
+
+	group->groupmask = groupmask;
+}
+
 static bool tmigr_init_root(struct tmigr_group *group, bool root_up)
 {
 	if (!group->parent && group != tmigr_root) {

@@ -1716,7 +1737,7 @@ static bool tmigr_init_root(struct tmigr_group *group, bool root_up)
 		 * to avoid accidents where yet another new top-level is
 		 * created in the future and made visible before this groupmask.
 		 */
-		group->groupmask = BIT(0);
+		tmigr_init_groupmask(group, BIT(0));
 		WARN_ON_ONCE(root_up);
 
 		return true;

@@ -1750,10 +1771,10 @@ static void tmigr_connect_child_parent(struct tmigr_group *child,
 		 * to the CPU going up has been accounted as the second child.
 		 */
 		WARN_ON_ONCE(parent->num_children != 2);
-		child->groupmask = BIT(0);
+		tmigr_init_groupmask(child, BIT(0));
 	} else {
 		/* Common case adding @child for the CPU going up to @parent. */
-		child->groupmask = BIT(parent->num_children++);
+		tmigr_init_groupmask(child, BIT(parent->num_children++));
 	}
 
 	/*

index 3f6c7a110e3c..0bf3a0e7d54c 100644
@@ -58,6 +58,7 @@ struct tmigr_event {
  *			tmigr_level_list; is required during setup when a
  *			new group needs to be connected to the existing
  *			hierarchy groups
+ * @want_low_migrator:	Group wants the lowest capacity migrator
  */
 struct tmigr_group {
 	raw_spinlock_t		lock;

@@ -71,6 +72,7 @@ struct tmigr_group {
 	unsigned int		num_children;
 	u8			groupmask;
 	struct list_head	list;
+	bool			want_low_migrator;
 };
 
 /**

index abb321b903c4..4f055fc08435 100755
@@ -33,8 +33,8 @@ class Node:
 	def set_lvl(self, lvl):
 		self.lvl = lvl
 
-	def set_numa(self, numa):
-		self.numa = numa
+	def set_family(self, family):
+		self.family = family
 
 	def set_num_children(self, num_children):
 		self.num_children = num_children

@@ -44,7 +44,7 @@ class Node:
 			parent_grp = self.parent.group
 		else:
 			parent_grp = "-"
-		return "Group: %s mask: %s parent: %s lvl: %d numa: %d num_children: %d" % (self.group, self.groupmask, parent_grp, self.lvl, self.numa, self.num_children)
+		return "Group: %s mask: %s parent: %s lvl: %d family: %d num_children: %d" % (self.group, self.groupmask, parent_grp, self.lvl, self.family, self.num_children)
 
 def get_node(group):
 	if group in Node.node_list:

@@ -55,32 +55,32 @@ def get_node(group):
 		return n
 
 def tmigr_connect_cpu_parent(ts, line):
-	s = re.search("tmigr_connect_cpu_parent: cpu=([0-9]+) groupmask=([0-9a-zA-Z]+) parent=([0-9a-zA-Z]+) lvl=([0-9]+) numa=([-]?[0-9]+) num_children=([0-9]+)", line)
+	s = re.search("tmigr_connect_cpu_parent: cpu=([0-9]+) groupmask=([0-9a-zA-Z]+) parent=([0-9a-zA-Z]+) lvl=([0-9]+) family=([-]?[0-9]+) num_children=([0-9]+)", line)
 	if s is None:
 		return False
-	(cpu, groupmask, parent, lvl, numa, num_children) = (int(s.group(1)), s.group(2), s.group(3), int(s.group(4)), int(s.group(5)), int(s.group(6)))
+	(cpu, groupmask, parent, lvl, family, num_children) = (int(s.group(1)), s.group(2), s.group(3), int(s.group(4)), int(s.group(5)), int(s.group(6)))
 	n = get_node(cpu)
 	p = get_node(parent)
 	n.set_parent(p)
 	n.set_groupmask(groupmask)
 	n.set_lvl(-1)
 	p.set_lvl(lvl)
-	p.set_numa(numa)
-	n.set_numa(numa)
+	p.set_family(family)
+	n.set_family(family)
 	p.set_num_children(num_children)
 	p.add_child(n)
 
 def tmigr_connect_child_parent(ts, line):
-	s = re.search("tmigr_connect_child_parent: group=([0-9a-zA-Z]+) groupmask=([0-9a-zA-Z]+) parent=([0-9a-zA-Z]+) lvl=([0-9]+) numa=([-]?[0-9]+) num_children=([0-9]+)", line)
+	s = re.search("tmigr_connect_child_parent: group=([0-9a-zA-Z]+) groupmask=([0-9a-zA-Z]+) parent=([0-9a-zA-Z]+) lvl=([0-9]+) family=([-]?[0-9]+) num_children=([0-9]+)", line)
 	if s is None:
 		return False
-	(group, groupmask, parent, lvl, numa, num_children) = (s.group(1), s.group(2), s.group(3), int(s.group(4)), int(s.group(5)), int(s.group(6)))
+	(group, groupmask, parent, lvl, family, num_children) = (s.group(1), s.group(2), s.group(3), int(s.group(4)), int(s.group(5)), int(s.group(6)))
 	n = get_node(group)
 	p = get_node(parent)
 	n.set_parent(p)
 	n.set_groupmask(groupmask)
 	p.set_lvl(lvl)
-	p.set_numa(numa)
+	p.set_family(family)
 	p.set_num_children(num_children)
 	p.add_child(n)
 

@@ -88,7 +88,7 @@ def populate(enode, node):
 	enode = enode.add_child(name = node.group)
 	enode.add_feature("groupmask", "m:%s" % node.groupmask)
 	enode.add_feature("lvl", "lvl:%d" % node.lvl)
-	enode.add_feature("numa", "node %d" % node.numa)
+	enode.add_feature("family", "family %d" % node.family)
 	enode.add_feature("num_children", "c=%d" % node.num_children)
 	for child in node.children:
 		populate(enode, child)

@@ -107,4 +107,4 @@ if __name__ == "__main__":
 		group = group.parent
 	root = Tree()
 	populate(root, group)
-	print(root.get_ascii(show_internal=True, attributes=["name", "numa", "lvl"]))
+	print(root.get_ascii(show_internal=True, attributes=["name", "family", "lvl"]))

2.53.0

Dump the mask of each group in order to verify they are correct and well sorted with respect to CPU capacities.

---
 scripts/timer_migration_tree.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 4f055fc08435..275b05fd6c1b 100755
@@ -90,6 +90,7 @@ def populate(enode, node):
 	enode.add_feature("lvl", "lvl:%d" % node.lvl)
 	enode.add_feature("family", "family %d" % node.family)
 	enode.add_feature("num_children", "c=%d" % node.num_children)
+	enode.add_feature("groupmask", "mask=%s" % node.groupmask)
 	for child in node.children:
 		populate(enode, child)
 

@@ -107,4 +108,4 @@ if __name__ == "__main__":
 		group = group.parent
 	root = Tree()
 	populate(root, group)
-	print(root.get_ascii(show_internal=True, attributes=["name", "family", "lvl"]))
+	print(root.get_ascii(show_internal=True, attributes=["name", "family", "lvl", "groupmask"]))

2.53.0

This seems to work for my orion o6, the resulting hierarchy is:

                           root lvl1
                      family=-1, want_low_migrator=1
          ________________|___________|___________|____________
         /                /           /           /            /
    mask 0x1         mask 0x2    mask 0x4    mask 0x8    mask 0x10
    grp279           grp866      grp905      grp984      grp1024
    CPUs 2,3,4,5     CPUs 8,9    CPUs 6,7    CPUs 10,11  CPUs 0,1

Honestly I think buckets should be fine here? One for each 128-capacity step (or even 256), which would also make these awkward Grace capacities all fall into one, which makes a lot of sense IMO?

Thanks for giving this a shot, here are the results, I tested this against
tmigr-off (which I've added ontop of series and capacity-unaware dd50a6e260ea "timers/migration: Revert per CPU capacity hierarchy")) In short: tmigr-off-600, capacity-unaware-600, series-600

idle-600s
----------

Overall idle states:

+------------------------+---------+---------+---------+--------------+
| Variant                | LPI-0   | LPI-1   | LPI-2   | LPI-1 + LPI-2|
+------------------------+---------+---------+---------+--------------+
| tmigr-off-600          | 3771.0s | 650.9s  | 2212.0s | 2862.9s      |
| capacity-unaware-600   | 2461.7s | 1245.3s | 2708.9s | 3954.3s      |
| series-600             | 2429.2s | 1260.7s | 2726.5s | 3987.2s      |
+------------------------+---------+---------+---------+--------------+

Delta vs tmigr-off-600:

+------------------------+----------+---------+---------+--------------+
| Variant                | LPI-0    | LPI-1   | LPI-2   | LPI-1 + LPI-2|
+------------------------+----------+---------+---------+--------------+
| capacity-unaware-600   | -1309.2s | 594.4s  | 497.0s  | 1091.4s      |
| series-600             | -1341.7s | 609.8s  | 514.5s  | 1124.3s      |
+------------------------+----------+---------+---------+--------------+

LPI-2 by capacity group:

+-------+----------------+------------------------+------------+
| Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
+-------+----------------+------------------------+------------+
| 279   | 805.3s         | 1007.3s                | 1003.5s    |
| 866   | 391.7s         | 483.5s                 | 477.3s     |
| 905   | 411.6s         | 466.1s                 | 501.8s     |
| 984   | 361.3s         | 468.3s                 | 458.9s     |
| 1024  | 242.2s         | 283.8s                 | 285.0s     |
+-------+----------------+------------------------+------------+

schbench-600s
-------------

Overall idle states:

+------------------------+--------+--------+--------+--------------+
| Variant                | LPI-0  | LPI-1  | LPI-2  | LPI-1 + LPI-2|
+------------------------+--------+--------+--------+--------------+
| tmigr-off-600          | 167.7s | 34.1s  | 22.3s  | 56.4s        |
| capacity-unaware-600   | 158.8s | 37.2s  | 28.1s  | 65.3s        |
| series-600             | 161.5s | 37.3s  | 26.3s  | 63.6s        |
+------------------------+--------+--------+--------+--------------+

Delta vs tmigr-off-600:

+------------------------+--------+-------+-------+--------------+
| Variant                | LPI-0  | LPI-1 | LPI-2 | LPI-1 + LPI-2|
+------------------------+--------+-------+-------+--------------+
| capacity-unaware-600   | -8.9s  | 3.1s  | 5.8s  | 8.9s         |
| series-600             | -6.2s  | 3.2s  | 4.0s  | 7.3s         |
+------------------------+--------+-------+-------+--------------+

LPI-2 by capacity group:

+-------+----------------+------------------------+------------+
| Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
+-------+----------------+------------------------+------------+
| 279   | 8.9s           | 10.6s                  | 10.3s      |
| 866   | 3.3s           | 4.8s                   | 4.3s       |
| 905   | 3.8s           | 4.6s                   | 4.3s       |
| 984   | 3.3s           | 4.8s                   | 4.7s       |
| 1024  | 3.0s           | 3.2s                   | 2.7s       |
+-------+----------------+------------------------+------------+

tbench-4clients-600s
--------------------

Overall idle states:

+------------------------+---------+---------+--------+--------------+
| Variant                | LPI-0   | LPI-1   | LPI-2  | LPI-1 + LPI-2|
+------------------------+---------+---------+--------+--------------+
| tmigr-off-600          | 3125.7s | 1395.7s | 425.9s | 1821.6s      |
| capacity-unaware-600   | 3009.4s | 1437.7s | 456.3s | 1894.0s      |
| series-600             | 2998.3s | 1436.3s | 453.6s | 1889.9s      |
+------------------------+---------+---------+--------+--------------+

Delta vs tmigr-off-600:

+------------------------+---------+--------+--------+--------------+
| Variant                | LPI-0   | LPI-1  | LPI-2  | LPI-1 + LPI-2|
+------------------------+---------+--------+--------+--------------+
| capacity-unaware-600   | -116.2s | 41.9s  | 30.5s  | 72.4s        |
| series-600             | -127.3s | 40.6s  | 27.7s  | 68.3s        |
+------------------------+---------+--------+--------+--------------+

LPI-2 by capacity group:

+-------+----------------+------------------------+------------+
| Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
+-------+----------------+------------------------+------------+
| 279   | 162.9s         | 179.3s                 | 180.5s     |
| 866   | 68.6s          | 73.4s                  | 71.9s      |
| 905   | 67.7s          | 69.5s                  | 71.8s      |
| 984   | 72.9s          | 75.5s                  | 71.5s      |
| 1024  | 53.7s          | 58.6s                  | 57.9s      |
+-------+----------------+------------------------+------------+

Seems the series does what it claims, I'd love to also hear about Sehee's results to the original regression (as honestly, capacity-unaware doesn't seem so bad for me in the first case, so maybe my testing has a blind spot here?)

It may be hard to find the right threshold. Also migrating from capX to capX-50 can still make a visible difference. So I'd rather keep all capacities separate (for now) and create more levels above in the hierarchy to group capacities if necessary. It should only create one more level if nr_cap > 8 and two levels if nr_cap > 64, but reaching that latter doesn't sound reasonable on the hardware, right? (famous last words).

Ok here it looks good.

But IIUC here it performs worse than before on LPI-2 ?

And overall worse in fact?

Same here?

Does it really?

Me too yes.

Thanks.

Frederic Weisbecker
SUSE Labs

So I'm only aware of systems with <=3 different uArch CPUs, there's a few with many more capacity classes, but these will be the same uArch with slightly different available frequencies, like in my example there's:
Quad Big Cortex®-A720 Up to 2.6GHz
Quad Medium Cortex®-A720 Up to 2.4GHz
Quad Little Cortex®-A520 1.8GHz
i.e. A720 and A520. I think as long as the capacity classes don't mix uArchs we should be fine...
We're far off from nr_cap > 64 FWIW but technically this can change even for existing silicon e.g. if FW notices the different cores age differently it could start reporting different frequencies from one boot to the next. That being said I agree nr_cap < 64 is okay for now...
I would've grouped them just because we do fundamentally want bigger capacity classes, right?

FWIW this should be the most important case.

In the sense that it seems to restore the benefit of tmigr in the first place, the rest would be interesting to hear from Sehee. I tried a little but found nothing to produce something that would explain such a big power regression. (Because the 'bad' 'little->big' migrations would have to absolutely dominate over the 'good' little->big (or good little->little and big->big) migrations).

Your interpretation on schbench and tbench capacity-unaware-600 vs series-600 is correct though, I'll go look some more into why that is...
It might just be the fragmentation of the bigs... (i.e. a migr grp984->grp1024 now disallowed but most likely it would still be 'good' and the fact we have many more big cores than little cores).

Hi,

I've been following the recent timer migration discussion and wanted to share numbers from a classic 4+4 big.LITTLE board I have on hand. cpuidle residency only; no power meter. Hope this is useful.

On 4+4 (all CPUs online): split raised LITTLE cpu-sleep under little-pin; idle and Christian-style load were flat across unaware / split / prefer-low.

1) Hardware

Radxa ROCK 4D, RK3576

  CPU0-3  Cortex-A53  cluster0  cpu_capacity=442   max 2016 MHz
  CPU4-7  Cortex-A72  cluster1  cpu_capacity=1024  max 2208 MHz

Two capacity classes, 4 CPUs each. This is not Christian's Orion O6 (five classes, many with only 2 CPUs). This report is 4+4 with all CPUs online; 4+1 / 2+2 topologies are planned after a gated-split patch (section 7).

Idle states (same names on all 8 CPUs):

    state0  WFI         latency=1     residency=1
    state1  cpu-sleep   latency=370   residency=900

No LPI-0/1/2. Tables below are cpu-sleep. WFI was only taken on idle-600 (unaware): ~0.2s A53 / ~1s A72 per cluster. The rest of the matrix was not collected once that looked negligible.

2) Kernels

Rootfs: Armbian_26.2.5_Radxa-rock-4d_resolute_edge_7.0.1_minimal. The stock 7.0.1 edge kernel was replaced. All three Images are next-20260902 built from that image's .config (same dtb / userspace).

unaware unmodified next-20260902

              7.3.0-rc1-next-20260902-next-rockpi4d
              dmesg: Timer migration: 1 hierarchy levels;
                     8 children per group; 1 crossnode level
              -> one LVL0 group, all 8 CPUs mixed

split 7.3.0-rc1-next-20260902-tmigr-split+
same tree; tmigr_get_capacity() returns real capacity (vanilla next keeps everyone at SCHED_CAPACITY_SCALE) boot dmesg line is unchanged (it is group-width, not topology). little-pin CPU1 residency moving from ~32s to ~120s is consistent with a 4-CPU LITTLE-only group. No timer_migration_tree.py dump: board had no network, so pip3/ete3 could not be installed. Sorry — no ASCII tree this time; can follow up if useful.

prefer-low 7.3.0-rc1-next-20260902-tmigr-prefer-low+

              Frederic RFT 0/6 rebased onto the same base
              git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
              timers/migration-capacity, HEAD: 28f02a6747ac4343f3596aaa3b749dea95577478
              dmesg: Timer migration: 2 hierarchy levels;
                     8 children per group; 1 crossfamily level
              -> two capacity families + a connecting level

Common: NO_HZ_IDLE, HZ=250, schedutil, kernel.timer_migration=1, ARM_PSCI_CPUIDLE + menu. No NO_HZ_FULL.

3) Workloads (N=3, 600s each unless noted)

idle-600 sleep 600
tbench tbench_srv; tbench -R 20 -t 600 4 127.0.0.1

               (Christian's -R 20; dbench 4.00 ~120s warmup; wall ~720s)

schbench schbench -m 2 -t 4 -r 600
little-pin four 4ms sleep loops (sleep 0.004) pinned to CPU0-3 for 600s
(Sehee-class load, tmigr on; not Sehee's tmigr=off case)

Residency is sum of /cpuidle/state1/time over the cluster, seconds. A53 max = 4 * wall; A72 max = 4 * wall. idle/schbench/pinned wall=600s => cluster max 2400s
tbench wall~720s => cluster max ~2880s

4) Main table -- per-run raw; mean is arithmetic mean of the three rows

cpu-sleep seconds = (after-before)/1e6 summed over the cluster.

idle-600 A53 / A72
unaware

    N=1  2395.7 / 2399.2
    N=2  2386.2 / 2400.2
    N=3  2367.6 / 2396.2
    mean 2383.2 / 2398.5

split

    N=1  2396.4 / 2398.4
    N=2  2398.1 / 2397.4
    N=3  2399.4 / 2372.5
    mean 2398.0 / 2389.4

prefer-low

    N=1  2394.8 / 2399.5
    N=2  2395.2 / 2380.0
    N=3  2377.9 / 2399.2
    mean 2389.3 / 2392.9

tbench A53 / A72 | throughput | max latency
unaware

    N=1  978.8 / 1985.2   14.4431 MB/s   4.444 ms
    N=2  982.6 / 1985.1   14.4405 MB/s   4.131 ms
    N=3  982.7 / 1983.9   14.4237 MB/s   3.817 ms
    mean 981.4 / 1984.7   14.4358 MB/s   4.131 ms

split

    N=1  976.7 / 1979.5   14.5214 MB/s   4.136 ms
    N=2  982.1 / 1981.2   14.5000 MB/s   4.017 ms
    N=3  982.2 / 1979.1   14.5176 MB/s   4.128 ms
    mean 980.3 / 1979.9   14.5130 MB/s   4.094 ms

prefer-low

    N=1  978.9 / 1984.1   14.4451 MB/s   4.268 ms
    N=2  978.7 / 1983.1   14.4437 MB/s   3.719 ms
    N=3  983.2 / 1985.0   14.3993 MB/s   3.876 ms
    mean 980.3 / 1984.1   14.4294 MB/s   3.954 ms

schbench A53 / A72 | wake p99 | req p99 | rps
unaware

    N=1  401.9 / 3.2   3996 us   56000 us   447.54
    N=2  396.3 / 3.2   4004 us   58176 us   433.53
    N=3  370.7 / 2.9   3996 us   62272 us   411.69
    mean 389.6 / 3.1   3999 us   58816 us   430.92

split

    N=1  420.5 / 4.0   3980 us   52544 us   469.75
    N=2  391.7 / 3.1   3996 us   56896 us   435.84
    N=3  367.7 / 3.1   3996 us   58176 us   417.24
    mean 393.3 / 3.4   3991 us   55872 us   440.94

prefer-low

    N=1  428.9 / 3.8   3980 us   51136 us   481.38
    N=2  425.1 / 3.3   3996 us   54208 us   463.08
    N=3  412.7 / 3.1   3996 us   56000 us   449.39
    mean 422.2 / 3.4   3991 us   53781 us   464.62

little-pin A53 / A72
unaware

    N=1  291.2 / 2377.0
    N=2  283.4 / 2376.4
    N=3  292.8 / 2376.8
    mean 289.1 / 2376.7

split

    N=1  346.3 / 2372.1
    N=2  408.2 / 2371.6
    N=3  381.8 / 2372.0
    mean 378.8 / 2371.9

prefer-low

    N=1  279.6 / 2379.6
    N=2  286.8 / 2377.7
    N=3  291.4 / 2377.5
    mean 285.9 / 2378.3

5) little-pin per-CPU cpu-sleep (s)

unaware

         cpu0   cpu1   cpu2   cpu3   cpu4   cpu5   cpu6   cpu7
    N=1  12.7   32.4   122.7  123.4  589.3  598.7  591.5  597.5
    N=2  11.7   31.7   120.8  119.1  588.3  596.1  595.4  596.6
    N=3  12.9   32.4   122.6  124.8  590.3  596.4  598.0  592.0

split

         cpu0   cpu1   cpu2   cpu3   cpu4   cpu5   cpu6   cpu7
    N=1  15.6   109.2  108.2  113.3  586.5  592.3  596.0  597.3
    N=2  16.1   132.4  129.2  130.5  585.2  589.5  598.2  598.7
    N=3  16.5   122.9  119.2  123.1  586.6  591.1  596.2  598.1

prefer-low

         cpu0   cpu1   cpu2   cpu3   cpu4   cpu5   cpu6   cpu7
    N=1  12.8   116.9  31.9   118.0  593.2  591.5  598.0  596.9
    N=2  12.4   121.5  31.5   121.3  592.5  595.4  590.7  599.1
    N=3  12.9   122.1  32.5   124.0  589.6  595.7  595.5  596.8

split: cpu1 109/132/123 vs unaware 32/32/32. prefer-low keeps two busy LITTLE CPUs (~13s / ~32s on cpu0+cpu2); split spreads sleep across cpu1-3 (~110-130s).

6) Reading the three columns on 4+4

little-pin, big (Sehee-class; tmigr on)
A72 2377 / 2372 / 2378 on all three variants (~594 s/CPU, ~5s below idle). Big deep idle not broken here; not Sehee's tmigr=off case. No A72 gap for split or prefer-low to recover.

little-pin, LITTLE: split vs O6
Christian's O6 failure mode (split hurts LITTLE) is opposite here:
A53 289 -> 379 on split (cpu1 recovered). prefer-low A53 286, unaware-like (cpu2 takes the ~32s slot).

last-idle big / migrator
No sign big held the global migrator under little-pin: A72 ~full cpu-sleep (~594 s/CPU). No migrator traces.

idle and load (prefer-low vs unaware)
Idle already ~100% cpu-sleep on unaware; prefer-low cannot win.
Load: tbench 14.44 / 14.51 / 14.43 MB/s; schbench wake p99 4.0 ms on all three (schbench keeps A72 busy, so low A72 residency there is expected). Not worse, not better. Same Christian O6 reading under load; here no idle prize either.

prefer-low: does it really? (Frederic, July 2026 thread)
Tree is up (2 levels, crossfamily). want_low_migrator not traced. little-pin shape (cpu0+cpu2 busy) matches a connected tree, not isolated split.

7) Next steps

Section 6 on this board: split raised LITTLE cpu-sleep on little-pin; prefer-low matched unaware; big stayed in cpu-sleep. Christian's O6 numbers went the other way on LITTLE when capacity classes are small.

I plan to patch tmigr_get_capacity() for a gated per-capacity split:
- online count in every capacity class >= 4 -> arch_scale_cpu_capacity()
- otherwise -> SCHED_CAPACITY_SCALE (same as unaware today)
- nohz_full still forces unaware

Re-test with the same Images base plus a gated-split build:
idle + little-pin + tbench on 4+4 here, then static 4+1 / 2+2 topologies (CPUs offlined before each run, held for the full 600s).

First try only: cutoff 4 is a starting guess (this board 4+4, O6 often 2). The first patch would count online CPUs at boot/setup only; no runtime hotplug tree rebuild, no split-big-only policy. prefer-low did not beat unaware; next step is gated split, not another prefer-low round.

I would appreciate feedback on cutoff choice, hotplug handling, and on section 8 gaps worth closing before a patch.

8) Not in this mail

- tmigr=off
- nohz_full
- tmigr_connect / handle_remote traces
- timer_migration_tree.py (no network on board for pip3/ete3; can follow up)
- 4+1 / 2+2 hotplug (after gated-split patch; not in this report)
- power (mW)

Thanks,
Liang Hao