Skip to contentSkip to conversation

timers: Allocate per-cpu tvec_base's statically

PATCH Superseded Viresh Kumar 3 participants 10 messages Mar 30, 2015 – Mar 31, 2015 High-resolution timers, timer wheel, clockevents

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Viresh Kumar

author
#0 lore raw

cover timers: Allocate per-cpu tvec_base's statically

Hi Ingo/Thomas,

Here are few cleanups around timer-core initialization. The first one is suggested by Peter Zijlstra [1] and the other two make few more changes in order to simplify it.

Please see if they look fine.

--
viresh

[1] http://lkml.iu.edu/hypermail/linux/kernel/1503.3/04091.html

Peter Zijlstra (1):
timer: Allocate per-cpu tvec_base's statically

Viresh Kumar (2):
timer: Limit the scope of __tvec_bases to init_timers_cpu()
timer: Don't initialize tvec_base on hotplug

 kernel/time/timer.c | 121 +++++++++++++++++++---------------------------------
 1 file changed, 45 insertions(+), 76 deletions(-)
Signature · 1 line
2.3.0.rc0.44.ga94655d
author
#1 lore raw

PATCH 1/3 timer: Allocate per-cpu tvec_base's statically

From: Peter Zijlstra <[email protected]>

Memory for tvec_base is allocated separately for boot CPU (statically) and non-boot CPUs (dynamically).

The reason is because __TIMER_INITIALIZER() needs to set ->base to a valid pointer (because we've made NULL special, hint: lock_timer_base()) and we cannot get a compile time pointer to per-cpu entries because we don't know where we'll map the section, even for the boot cpu.

This can be simplified a bit by statically allocating per-cpu memory. The only disadvantage is that memory for one of the structures will stay unused, i.e. for the boot CPU, which uses boot_tvec_bases.

This will also guarantee that tvec_base is cacheline aligned. Even though tvec_base has ____cacheline_aligned stuck on, kzalloc_node() does not actually respect that (but guarantees a minimum u64 alignment).

Signed-off-by Peter Zijlstra <[email protected]>
Signed-off-by Viresh Kumar <[email protected]>
---
 kernel/time/timer.c | 36 ++++++++----------------------------
 1 file changed, 8 insertions(+), 28 deletions(-)
kernel/time/timer.c +8 -28
index 2d3f5c504939..6e8220ec8a62 100644
@@ -93,6 +93,7 @@ struct tvec_base {
 struct tvec_base boot_tvec_bases;
 EXPORT_SYMBOL(boot_tvec_bases);
 static DEFINE_PER_CPU(struct tvec_base *, tvec_bases) = &boot_tvec_bases;
+static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
 
 /* Functions below help us manage 'deferrable' flag */
 static inline unsigned int tbase_get_deferrable(struct tvec_base *base)

@@ -1534,46 +1535,25 @@ EXPORT_SYMBOL(schedule_timeout_uninterruptible);
 
 static int init_timers_cpu(int cpu)
 {
-	int j;
-	struct tvec_base *base;
+	struct tvec_base *base = per_cpu(tvec_bases, cpu);
 	static char tvec_base_done[NR_CPUS];
+	int j;
 
 	if (!tvec_base_done[cpu]) {
 		static char boot_done;
 
-		if (boot_done) {
-			/*
-			 * The APs use this path later in boot
-			 */
-			base = kzalloc_node(sizeof(*base), GFP_KERNEL,
-					    cpu_to_node(cpu));
-			if (!base)
-				return -ENOMEM;
-
-			/* Make sure tvec_base has TIMER_FLAG_MASK bits free */
-			if (WARN_ON(base != tbase_get_base(base))) {
-				kfree(base);
-				return -ENOMEM;
-			}
-			per_cpu(tvec_bases, cpu) = base;
+		if (!boot_done) {
+			boot_done = 1; /* skip the boot cpu */
 		} else {
-			/*
-			 * This is for the boot CPU - we use compile-time
-			 * static initialisation because per-cpu memory isn't
-			 * ready yet and because the memory allocators are not
-			 * initialised either.
-			 */
-			boot_done = 1;
-			base = &boot_tvec_bases;
+			base = per_cpu_ptr(&__tvec_bases, cpu);
+			per_cpu(tvec_bases, cpu) = base;
 		}
+
 		spin_lock_init(&base->lock);
 		tvec_base_done[cpu] = 1;
 		base->cpu = cpu;
-	} else {
-		base = per_cpu(tvec_bases, cpu);
 	}
 
-
 	for (j = 0; j < TVN_SIZE; j++) {
 		INIT_LIST_HEAD(base->tv5.vec + j);
 		INIT_LIST_HEAD(base->tv4.vec + j);
Signature · 1 line
2.3.0.rc0.44.ga94655d
#2 lore raw

Re: [PATCH 1/3] timer: Allocate per-cpu tvec_base's statically

Quoted 41 lines · Viresh Kumar

From: Peter Zijlstra <[email protected]>

Memory for tvec_base is allocated separately for boot CPU (statically) and non-boot CPUs (dynamically).

The reason is because __TIMER_INITIALIZER() needs to set ->base to a valid pointer (because we've made NULL special, hint: lock_timer_base()) and we cannot get a compile time pointer to per-cpu entries because we don't know where we'll map the section, even for the boot cpu.

This can be simplified a bit by statically allocating per-cpu memory. The only disadvantage is that memory for one of the structures will stay unused, i.e. for the boot CPU, which uses boot_tvec_bases.

This will also guarantee that tvec_base is cacheline aligned. Even though tvec_base has ____cacheline_aligned stuck on, kzalloc_node() does not actually respect that (but guarantees a minimum u64 alignment).

Signed-off-by Peter Zijlstra <[email protected]>
Signed-off-by Viresh Kumar <[email protected]>
---
 kernel/time/timer.c | 36 ++++++++----------------------------
 1 file changed, 8 insertions(+), 28 deletions(-)
kernel/time/timer.c +1 -1
index 2d3f5c504939..6e8220ec8a62 100644
@@ -93,6 +93,7 @@ struct tvec_base {
 struct tvec_base boot_tvec_bases;
 EXPORT_SYMBOL(boot_tvec_bases);
 static DEFINE_PER_CPU(struct tvec_base *, tvec_bases) = &boot_tvec_bases;
+static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
 
 /* Functions below help us manage 'deferrable' flag */
 static inline unsigned int tbase_get_deferrable(struct tvec_base *base)

@@ -1534,46 +1535,25 @@ EXPORT_SYMBOL(schedule_timeout_uninterruptible);
 
 static int init_timers_cpu(int cpu)
 {
-	int j;

kernel/time/timer.c EXPORT_SYMBOL(schedule_timeout_uninterruptible); · line 1546

Quoted 10 lines
kernel/time/timer.c +4 -1
-	struct tvec_base *base;
+	struct tvec_base *base = per_cpu(tvec_bases, cpu);
 	static char tvec_base_done[NR_CPUS];
+	int j;
 
 	if (!tvec_base_done[cpu]) {
 		static char boot_done;
 
+		if (!boot_done) {
+			boot_done = 1; /* skip the boot cpu */

So it would be a lot more descriptive to name this flag 'boot_cpu_skipped'?

kernel/time/timer.c EXPORT_SYMBOL(schedule_timeout_uninterruptible); · line 1555

Quoted 9 lines
kernel/time/timer.c +3 -0
 		} else {
+			base = per_cpu_ptr(&__tvec_bases, cpu);
+			per_cpu(tvec_bases, cpu) = base;
 		}
+
 		spin_lock_init(&base->lock);
 		tvec_base_done[cpu] = 1;
 		base->cpu = cpu;
 	}

Also, I'd put a description about the PER_CPU background into comments as well, because it's not obvious at first sight at all what the whole (boot_tvec_bases, tvec_bases, __tvec_bases) dance does.

Thanks,

Ingo

author
#3 lore raw

Re: [PATCH 1/3] timer: Allocate per-cpu tvec_base's statically

Quoted 10 lines · Ingo Molnar

kernel/time/timer.c EXPORT_SYMBOL(schedule_timeout_uninterruptible); · line 1546

Quoted 4 lines · Viresh Kumar

From: Peter Zijlstra <[email protected]>

kernel/time/timer.c +2 -0
+             if (!boot_done) {
+                     boot_done = 1; /* skip the boot cpu */

So it would be a lot more descriptive to name this flag 'boot_cpu_skipped'?

Yes.

Quoted 3 lines

Also, I'd put a description about the PER_CPU background into comments as well, because it's not obvious at first sight at all what the whole (boot_tvec_bases, tvec_bases, __tvec_bases) dance does.

Yeah, so I did that with a one liner in the last patch, but that doesn't look good enough. I will try to do something better in V2.

Thanks for your reviews.

author
#4 lore raw

PATCH 2/3 timer: Limit the scope of __tvec_bases to init_timers_cpu()

In reply to Viresh Kumar

per-cpu variable '__tvec_bases' is used for all CPUs except the boot CPU, which uses 'boot_tvec_bases'. Because of this special case, we shouldn't make direct references to __tvec_bases from timer core as there are chances of using the wrong instance for boot CPU.

To force that, lets move __tvec_bases to the only routine which can use it directly (to set tvec_bases).

Signed-off-by Viresh Kumar <[email protected]>
---
 kernel/time/timer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/timer.c +1 -1
index 6e8220ec8a62..40918a5d3e1d 100644
@@ -93,7 +93,6 @@ struct tvec_base {
 struct tvec_base boot_tvec_bases;
 EXPORT_SYMBOL(boot_tvec_bases);
 static DEFINE_PER_CPU(struct tvec_base *, tvec_bases) = &boot_tvec_bases;
-static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
 
 /* Functions below help us manage 'deferrable' flag */
 static inline unsigned int tbase_get_deferrable(struct tvec_base *base)

@@ -1537,6 +1536,7 @@ static int init_timers_cpu(int cpu)
 {
 	struct tvec_base *base = per_cpu(tvec_bases, cpu);
 	static char tvec_base_done[NR_CPUS];
+	static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
 	int j;
 
 	if (!tvec_base_done[cpu]) {
Signature · 1 line
2.3.0.rc0.44.ga94655d
author
#5 lore raw

PATCH 3/3 timer: Don't initialize tvec_base on hotplug

In reply to Viresh Kumar

There is no need to call init_timers_cpu() on every cpu hotplug event, there is not much we need to reset.

- Timer-lists are already empty at the end of migrate_timers().
- timer_jiffies will be refreshed while adding a new timer, after the CPU is online again.
- active_timers and all_timers can be reset from migrate_timers().

Signed-off-by Viresh Kumar <[email protected]>
---
 kernel/time/timer.c | 101 +++++++++++++++++++++++-----------------------------
 1 file changed, 45 insertions(+), 56 deletions(-)
kernel/time/timer.c +45 -56
index 40918a5d3e1d..aa02d49b3748 100644
@@ -1532,44 +1532,6 @@ signed long __sched schedule_timeout_uninterruptible(signed long timeout)
 }
 EXPORT_SYMBOL(schedule_timeout_uninterruptible);
 
-static int init_timers_cpu(int cpu)
-{
-	struct tvec_base *base = per_cpu(tvec_bases, cpu);
-	static char tvec_base_done[NR_CPUS];
-	static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
-	int j;
-
-	if (!tvec_base_done[cpu]) {
-		static char boot_done;
-
-		if (!boot_done) {
-			boot_done = 1; /* skip the boot cpu */
-		} else {
-			base = per_cpu_ptr(&__tvec_bases, cpu);
-			per_cpu(tvec_bases, cpu) = base;
-		}
-
-		spin_lock_init(&base->lock);
-		tvec_base_done[cpu] = 1;
-		base->cpu = cpu;
-	}
-
-	for (j = 0; j < TVN_SIZE; j++) {
-		INIT_LIST_HEAD(base->tv5.vec + j);
-		INIT_LIST_HEAD(base->tv4.vec + j);
-		INIT_LIST_HEAD(base->tv3.vec + j);
-		INIT_LIST_HEAD(base->tv2.vec + j);
-	}
-	for (j = 0; j < TVR_SIZE; j++)
-		INIT_LIST_HEAD(base->tv1.vec + j);
-
-	base->timer_jiffies = jiffies;
-	base->next_timer = base->timer_jiffies;
-	base->active_timers = 0;
-	base->all_timers = 0;
-	return 0;
-}
-
 #ifdef CONFIG_HOTPLUG_CPU
 static void migrate_timer_list(struct tvec_base *new_base, struct list_head *head)
 {

@@ -1611,6 +1573,9 @@ static void migrate_timers(int cpu)
 		migrate_timer_list(new_base, old_base->tv5.vec + i);
 	}
 
+	old_base->active_timers = 0;
+	old_base->all_timers = 0;
+
 	spin_unlock(&old_base->lock);
 	spin_unlock_irq(&new_base->lock);
 	put_cpu_var(tvec_bases);

@@ -1620,25 +1585,16 @@ static void migrate_timers(int cpu)
 static int timer_cpu_notify(struct notifier_block *self,
 				unsigned long action, void *hcpu)
 {
-	long cpu = (long)hcpu;
-	int err;
-
-	switch(action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
-		err = init_timers_cpu(cpu);
-		if (err < 0)
-			return notifier_from_errno(err);
-		break;
 #ifdef CONFIG_HOTPLUG_CPU
+	switch (action) {
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
-		migrate_timers(cpu);
+		migrate_timers((long)hcpu);
 		break;
-#endif
 	default:
 		break;
 	}
+#endif
 	return NOTIFY_OK;
 }
 

@@ -1646,18 +1602,51 @@ static struct notifier_block timers_nb = {
 	.notifier_call	= timer_cpu_notify,
 };
 
+static void __init init_timer_cpu(struct tvec_base *base, int cpu)
+{
+	int j;
 
-void __init init_timers(void)
+	base->cpu = cpu;
+	per_cpu(tvec_bases, cpu) = base;
+	spin_lock_init(&base->lock);
+
+	for (j = 0; j < TVN_SIZE; j++) {
+		INIT_LIST_HEAD(base->tv5.vec + j);
+		INIT_LIST_HEAD(base->tv4.vec + j);
+		INIT_LIST_HEAD(base->tv3.vec + j);
+		INIT_LIST_HEAD(base->tv2.vec + j);
+	}
+	for (j = 0; j < TVR_SIZE; j++)
+		INIT_LIST_HEAD(base->tv1.vec + j);
+
+	base->timer_jiffies = jiffies;
+	base->next_timer = base->timer_jiffies;
+}
+
+static void __init init_timer_cpus(void)
 {
-	int err;
+	/* Used for all CPUs, except boot CPU (which uses boot_tvec_bases) */
+	static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
+	struct tvec_base *base;
+	int local_cpu = smp_processor_id();
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		if (likely(cpu != local_cpu))
+			base = per_cpu_ptr(&__tvec_bases, cpu);
+		else
+			base = &boot_tvec_bases;
+
+		init_timer_cpu(base, cpu);
+	}
+}
 
+void __init init_timers(void)
+{
 	/* ensure there are enough low bits for flags in timer->base pointer */
 	BUILD_BUG_ON(__alignof__(struct tvec_base) & TIMER_FLAG_MASK);
 
-	err = timer_cpu_notify(&timers_nb, (unsigned long)CPU_UP_PREPARE,
-			       (void *)(long)smp_processor_id());
-	BUG_ON(err != NOTIFY_OK);
-
+	init_timer_cpus();
 	init_timer_stats();
 	register_cpu_notifier(&timers_nb);
 	open_softirq(TIMER_SOFTIRQ, run_timer_softirq);
Signature · 1 line
2.3.0.rc0.44.ga94655d
#6 lore raw
In reply to Viresh Kumar
Quoted 5 lines · Viresh Kumar

Hi Ingo/Thomas,

Here are few cleanups around timer-core initialization. The first one is suggested by Peter Zijlstra [1] and the other two make few more changes in order to simplify it.

I did the below on top; I'll sit on these patches for a wee while in the hope of the blackfin maintainer explaining his reasons for wrecking ____cacheline_aligned.

---
Subject: timer: Further simplify SMP and HOTPLUG logic
From: Peter Zijlstra <[email protected]>
Date: Mon Mar 30 10:09:19 CEST 2015

Remove one CONFIG_HOTPLUG_CPU #ifdef in trade for introducing one CONFIG_SMP #ifdef.

The CONFIG_SMP ifdef avoids declaring the per-cpu __tvec_bases storage on UP systems since they already have boot_tvec_bases.

Also (re)add a runtime check on the base alignment -- for the paranoid amongst us :-)

Signed-off-by Peter Zijlstra (Intel) <[email protected]>
---
 kernel/time/timer.c |   28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)
kernel/time/timer.c +13 -15
@@ -1580,12 +1580,10 @@ static void migrate_timers(int cpu)
 	spin_unlock_irq(&new_base->lock);
 	put_cpu_var(tvec_bases);
 }
-#endif /* CONFIG_HOTPLUG_CPU */
 
 static int timer_cpu_notify(struct notifier_block *self,
 				unsigned long action, void *hcpu)
 {
-#ifdef CONFIG_HOTPLUG_CPU
 	switch (action) {
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:

@@ -1594,18 +1592,17 @@ static int timer_cpu_notify(struct notif
 	default:
 		break;
 	}
-#endif
+
 	return NOTIFY_OK;
 }
-
-static struct notifier_block timers_nb = {
-	.notifier_call	= timer_cpu_notify,
-};
+#endif /* CONFIG_HOTPLUG_CPU */
 
 static void __init init_timer_cpu(struct tvec_base *base, int cpu)
 {
 	int j;
 
+	BUG_ON(base != tbase_get_base(base));
+
 	base->cpu = cpu;
 	per_cpu(tvec_bases, cpu) = base;
 	spin_lock_init(&base->lock);

@@ -1625,18 +1622,19 @@ static void __init init_timer_cpu(struct
 
 static void __init init_timer_cpus(void)
 {
-	/* Used for all CPUs, except boot CPU (which uses boot_tvec_bases) */
-	static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
-	struct tvec_base *base;
 	int local_cpu = smp_processor_id();
+	struct tvec_base *base;
 	int cpu;
 
 	for_each_possible_cpu(cpu) {
-		if (likely(cpu != local_cpu))
-			base = per_cpu_ptr(&__tvec_bases, cpu);
-		else
+		if (cpu == local_cpu)
 			base = &boot_tvec_bases;
-
+#ifdef CONFIG_SMP
+		else {
+			static DEFINE_PER_CPU(struct tvec_base, __tvec_bases);
+			base = per_cpu_ptr(&__tvec_bases, cpu);
+		}
+#endif
 		init_timer_cpu(base, cpu);
 	}
 }

@@ -1648,7 +1646,7 @@ void __init init_timers(void)
 
 	init_timer_cpus();
 	init_timer_stats();
-	register_cpu_notifier(&timers_nb);
+	cpu_notifier(timer_cpu_notify, 0);
 	open_softirq(TIMER_SOFTIRQ, run_timer_softirq);
 }
author Reviewed-by
#7 lore raw
Quoted 3 lines · Peter Zijlstra

I did the below on top; I'll sit on these patches for a wee while in the hope of the blackfin maintainer explaining his reasons for wrecking ____cacheline_aligned.

Two things I wanted to mention:

1.) We may need to drop the patch 2/3 as that breaks build for xtensa and tile.

kernel/time/timer.c: In function 'init_timers_cpu':
kernel/time/timer.c:1539:1: error: section attribute cannot be
specified for local variables
kernel/time/timer.c:1539:1: error: section attribute cannot be
specified for local variables

Quoted 1 line
Quoted 1 line

kernel/time/timer.c:1539:1: error: declaration of '__pcpu_unique___tvec_bases' with no linkage follows extern declaration

kernel/time/timer.c:1539:1: note: previous declaration of
'__pcpu_unique___tvec_bases' was here
kernel/time/timer.c:1539:9: error: section attribute cannot be
specified for local variables
kernel/time/timer.c:1539:9: error: section attribute cannot be
specified for local variables
kernel/time/timer.c:1539:9: error: weak declaration of
'__tvec_bases' must be public
kernel/time/timer.c:1539:9: error: declaration of '__tvec_bases'
with no linkage follows extern declaration
kernel/time/timer.c:1539:9: note: previous declaration of
'__tvec_bases' was here

Can these be fixed somehow ?

2.) We probably don't need to wait for the blackfin guys as we haven't broken anything yet. It will be broken only once we start using an extra bit from base pointer, which will be done by a separate patch trying to migrate running timers.

Quoted 11 lines

---
Subject: timer: Further simplify SMP and HOTPLUG logic
From: Peter Zijlstra <[email protected]>
Date: Mon Mar 30 10:09:19 CEST 2015

Remove one CONFIG_HOTPLUG_CPU #ifdef in trade for introducing one CONFIG_SMP #ifdef.

The CONFIG_SMP ifdef avoids declaring the per-cpu __tvec_bases storage on UP systems since they already have boot_tvec_bases.

Quoted 7 lines

Also (re)add a runtime check on the base alignment -- for the paranoid amongst us :-)

Signed-off-by Peter Zijlstra (Intel) <[email protected]>
---
 kernel/time/timer.c |   28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)
Reviewed-by Viresh Kumar <[email protected]>
#8 lore raw
Quoted 7 lines · Viresh Kumar
Quoted 3 lines · Peter Zijlstra

I did the below on top; I'll sit on these patches for a wee while in the hope of the blackfin maintainer explaining his reasons for wrecking ____cacheline_aligned.

Two things I wanted to mention:

Quoted 7 lines

1.) We may need to drop the patch 2/3 as that breaks build for xtensa and tile.

kernel/time/timer.c: In function 'init_timers_cpu':
kernel/time/timer.c:1539:1: error: section attribute cannot be
specified for local variables

Can these be fixed somehow ?

Hmm, is your xtensa/tile compiler different from your others? But no, I think we need to move it back to global scope. Lemme go do that.

author
#9 lore raw
Quoted 2 lines · Peter Zijlstra

Hmm, is your xtensa/tile compiler different from your others? But no, I think we need to move it back to global scope. Lemme go do that.

These were generated by Fenguang's build-bot.