sched: add notifier for cross-cpu migrations

PATCH Applied Peter Zijlstra 4 participants 11 messages Jul 10, 2013 – Aug 23, 2013

Subsystems (2)

x86 Scheduler

lore

Share & tools

lore

Peter Zijlstra

author maintainer

2013-07-10 10:35 UTC #0 lore raw

Hi,

I just stumbled over commit 582b336ec2 and had a massive WTF moment.

The Changelog -- empty!
The Implementation -- complete crap!

fail^2.

A git grep later I find its x86_64 paravirt only.. for this we add unconditional crap to the scheduler?

At the _very_ least this should have been wrapped in a static_key and the changelog should have given some clue as to the what and why of this code.

/me grumpy

Ingo Molnar

2013-07-10 14:39 UTC #1 lore raw

Quoted 7 lines · Peter Zijlstra

Hi,
I just stumbled over commit 582b336ec2 and had a massive WTF moment.
The Changelog -- empty!
The Implementation -- complete crap!

Quoted 8 lines

fail^2.
A git grep later I find its x86_64 paravirt only.. for this we add unconditional crap to the scheduler?
At the _very_ least this should have been wrapped in a static_key and the changelog should have given some clue as to the what and why of this code.

My bad for letting it slip through...

Marcelo, mind implementing the suggestions from Peter?

Thanks,

Ingo

Peter Zijlstra

author maintainer

2013-07-10 14:53 UTC #2 lore raw

Quoted 15 lines · Ingo Molnar

Quoted 12 lines · Peter Zijlstra
Hi,
I just stumbled over commit 582b336ec2 and had a massive WTF moment.
The Changelog -- empty!
The Implementation -- complete crap!
fail^2.
A git grep later I find its x86_64 paravirt only.. for this we add unconditional crap to the scheduler?

Quoted 7 lines

Quoted 3 lines
At the _very_ least this should have been wrapped in a static_key and the changelog should have given some clue as to the what and why of this code.
My bad for letting it slip through...
Marcelo, mind implementing the suggestions from Peter?

So ideally we'd kill the entire notifier, notifiers make it far too easy for others to use -- and I don't want silent users of stuff like this.

Ideally I'd see a direct callback:

  pvclock_migration_callback(t, new_cpu);

That compiles to 'do { } while (0);' for kernels without CONFIG_PARAVIRT. Then for paravirt it would use a static_key to get a real callback only if there's a pvclock user.

Something like:

#define pvclock_migration_callback(_t, _cpu) \

do {								\
	if (static_key_false(&pvclock_key))			\
		__pvclock_migration_callback((_t), (_cpu));	\
} while (0)

NOTE: static_key_false() doesn't test false, it assumes the default is false and makes the function call the out-of-line jump -- horridly confusing function name.

Your pvclock muck would do:

static_key_slow_inc() -- for every new user and,
static_key_slow_dec() -- for every user gone.

Its a reference count scheme, so that when there's no users there is only a 5 byte nop.

Marcelo Tosatti

2013-07-11 01:23 UTC #3 lore raw

PATCH 1/1 remove sched notifier for cross-cpu migrations

In reply to Peter Zijlstra

Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:

1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.

Which is sufficient to guarantee nanoseconds counter is calculated properly.

Signed-off-by Marcelo Tosatti <[email protected]>

arch/x86/include/asm/pvclock.h +0 -1

index 109a9dd..be8269b 100644
@@ -93,7 +93,6 @@ unsigned __pvclock_read_cycles(const struct pvclock_vcpu_time_info *src,
 
 struct pvclock_vsyscall_time_info {
 	struct pvclock_vcpu_time_info pvti;
-	u32 migrate_count;
 } __attribute__((__aligned__(SMP_CACHE_BYTES)));
 
 #define PVTI_SIZE sizeof(struct pvclock_vsyscall_time_info)

arch/x86/kernel/pvclock.c +0 ~~-44~~

63 lines not on this page · raw message

arch/x86/vdso/vclock_gettime.c +8 -8

index c74436e..72074d5 100644
@@ -85,15 +85,18 @@ static notrace cycle_t vread_pvclock(int *mode)
 	cycle_t ret;
 	u64 last;
 	u32 version;
-	u32 migrate_count;
 	u8 flags;
 	unsigned cpu, cpu1;
 
 
 	/*
-	 * When looping to get a consistent (time-info, tsc) pair, we
-	 * also need to deal with the possibility we can switch vcpus,
-	 * so make sure we always re-fetch time-info for the current vcpu.
+	 * Note: hypervisor must guarantee that:
+	 * 1. cpu ID number maps 1:1 to per-CPU pvclock time info.
+	 * 2. that per-CPU pvclock time info is updated if the
+	 *    underlying CPU changes.
+	 * 3. that version is increased whenever underlying CPU
+	 *    changes.
+	 *
 	 */
 	do {
 		cpu = __getcpu() & VGETCPU_CPU_MASK;

@@ -104,8 +107,6 @@ static notrace cycle_t vread_pvclock(int *mode)
 
 		pvti = get_pvti(cpu);
 
-		migrate_count = pvti->migrate_count;
-
 		version = __pvclock_read_cycles(&pvti->pvti, &ret, &flags);
 
 		/*

@@ -117,8 +118,7 @@ static notrace cycle_t vread_pvclock(int *mode)
 		cpu1 = __getcpu() & VGETCPU_CPU_MASK;
 	} while (unlikely(cpu != cpu1 ||
 			  (pvti->pvti.version & 1) ||
-			  pvti->pvti.version != version ||
-			  pvti->migrate_count != migrate_count));
+			  pvti->pvti.version != version));
 
 	if (unlikely(!(flags & PVCLOCK_TSC_STABLE_BIT)))
 		*mode = VCLOCK_NONE;

include/linux/sched.h +0 -8

index 178a8d9..217ce4b 100644
@@ -107,14 +107,6 @@ extern unsigned long this_cpu_load(void);
 extern void calc_global_load(unsigned long ticks);
 extern void update_cpu_load_nohz(void);
 
-/* Notifier for when a task gets migrated to a new CPU */
-struct task_migration_notifier {
-	struct task_struct *task;
-	int from_cpu;
-	int to_cpu;
-};
-extern void register_task_migration_notifier(struct notifier_block *n);
-
 extern unsigned long get_parent_ip(unsigned long addr);
 
 extern void dump_cpu_task(int cpu);

kernel/sched/core.c +0 ~~-15~~

index e8b3350..122e499 100644
@@ -974,13 +974,6 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)
 		rq->skip_clock_update = 1;
 }
 
-static ATOMIC_NOTIFIER_HEAD(task_migration_notifier);
-
-void register_task_migration_notifier(struct notifier_block *n)
-{
-	atomic_notifier_chain_register(&task_migration_notifier, n);
-}
-
 #ifdef CONFIG_SMP
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {

@@ -1011,18 +1004,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	trace_sched_migrate_task(p, new_cpu);
 
 	if (task_cpu(p) != new_cpu) {
-		struct task_migration_notifier tmn;
-
 		if (p->sched_class->migrate_task_rq)
 			p->sched_class->migrate_task_rq(p, new_cpu);
 		p->se.nr_migrations++;
 		perf_sw_event(PERF_COUNT_SW_CPU_MIGRATIONS, 1, NULL, 0);
-
-		tmn.task = p;
-		tmn.from_cpu = task_cpu(p);
-		tmn.to_cpu = new_cpu;
-
-		atomic_notifier_call_chain(&task_migration_notifier, 0, &tmn);
 	}
 
 	__set_task_cpu(p, new_cpu);

Marcelo Tosatti

2013-07-12 19:28 UTC #5 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

Quoted 6 lines · Gleb Natapov

Quoted 5 lines · Marcelo Tosatti
Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:

Quoted 8 lines

Quoted 6 lines
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.
Is this the case with KVM though? IIRC KVM does not updates kvmclock on vcpu migration if host has synchronized TSC.

KVM does not update kvmclock on vcpu->pcpu migration if the master clock scheme is used, which depends on synchronized TSC. In that case, it is safe to migrate a thread/vcpu to another cpu without updating kvmclock area (because all vcpus contain the same value as system timestamp and different tsc_timestamps, but all (tsc_timestamps) usable as deltas on any given pcpu.

So the 3 items are necessary in the general case of hypervisor maintaining distinct kvmclock vcpu areas (copy&pasted the comment from the code comment).

Peter Zijlstra

author maintainer Acked-by on v1 1/1

2013-07-11 09:53 UTC #6 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

In reply to Marcelo Tosatti

Quoted 11 lines · Marcelo Tosatti

Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.

Quoted 4 lines

Which is sufficient to guarantee nanoseconds counter is calculated properly.
Signed-off-by Marcelo Tosatti <[email protected]>

Awesome, thanks!

Acked-by Peter Zijlstra <[email protected]>

Gleb Natapov

Applied

2013-07-14 11:17 UTC #7 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

In reply to Marcelo Tosatti

Quoted 11 lines · Marcelo Tosatti

Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.

Quoted 5 lines

Which is sufficient to guarantee nanoseconds counter is calculated properly.
Signed-off-by Marcelo Tosatti <[email protected]>

Applied, thanks.

Quoted 174 lines

arch/x86/include/asm/pvclock.h +0 -1

index 109a9dd..be8269b 100644
@@ -93,7 +93,6 @@ unsigned __pvclock_read_cycles(const struct pvclock_vcpu_time_info *src,
 
 struct pvclock_vsyscall_time_info {
 	struct pvclock_vcpu_time_info pvti;
-	u32 migrate_count;
 } __attribute__((__aligned__(SMP_CACHE_BYTES)));
 
 #define PVTI_SIZE sizeof(struct pvclock_vsyscall_time_info)

arch/x86/kernel/pvclock.c +0 ~~-44~~

index 2cb9470..a16bae3 100644
@@ -128,46 +128,7 @@ void pvclock_read_wallclock(struct pvclock_wall_clock *wall_clock,
 	set_normalized_timespec(ts, now.tv_sec, now.tv_nsec);
 }
 
-static struct pvclock_vsyscall_time_info *pvclock_vdso_info;
-
-static struct pvclock_vsyscall_time_info *
-pvclock_get_vsyscall_user_time_info(int cpu)
-{
-	if (!pvclock_vdso_info) {
-		BUG();
-		return NULL;
-	}
-
-	return &pvclock_vdso_info[cpu];
-}
-
-struct pvclock_vcpu_time_info *pvclock_get_vsyscall_time_info(int cpu)
-{
-	return &pvclock_get_vsyscall_user_time_info(cpu)->pvti;
-}
-
 #ifdef CONFIG_X86_64
-static int pvclock_task_migrate(struct notifier_block *nb, unsigned long l,
-			        void *v)
-{
-	struct task_migration_notifier *mn = v;
-	struct pvclock_vsyscall_time_info *pvti;
-
-	pvti = pvclock_get_vsyscall_user_time_info(mn->from_cpu);
-
-	/* this is NULL when pvclock vsyscall is not initialized */
-	if (unlikely(pvti == NULL))
-		return NOTIFY_DONE;
-
-	pvti->migrate_count++;
-
-	return NOTIFY_DONE;
-}
-
-static struct notifier_block pvclock_migrate = {
-	.notifier_call = pvclock_task_migrate,
-};
-
 /*
  * Initialize the generic pvclock vsyscall state.  This will allocate
  * a/some page(s) for the per-vcpu pvclock information, set up a

@@ -181,17 +142,12 @@ int __init pvclock_init_vsyscall(struct pvclock_vsyscall_time_info *i,
 
 	WARN_ON (size != PVCLOCK_VSYSCALL_NR_PAGES*PAGE_SIZE);
 
-	pvclock_vdso_info = i;
-
 	for (idx = 0; idx <= (PVCLOCK_FIXMAP_END-PVCLOCK_FIXMAP_BEGIN); idx++) {
 		__set_fixmap(PVCLOCK_FIXMAP_BEGIN + idx,
 			     __pa(i) + (idx*PAGE_SIZE),
 			     PAGE_KERNEL_VVAR);
 	}
 
-
-	register_task_migration_notifier(&pvclock_migrate);
-
 	return 0;
 }
 #endif

arch/x86/vdso/vclock_gettime.c +8 -8

index c74436e..72074d5 100644
@@ -85,15 +85,18 @@ static notrace cycle_t vread_pvclock(int *mode)
 	cycle_t ret;
 	u64 last;
 	u32 version;
-	u32 migrate_count;
 	u8 flags;
 	unsigned cpu, cpu1;
 
 
 	/*
-	 * When looping to get a consistent (time-info, tsc) pair, we
-	 * also need to deal with the possibility we can switch vcpus,
-	 * so make sure we always re-fetch time-info for the current vcpu.
+	 * Note: hypervisor must guarantee that:
+	 * 1. cpu ID number maps 1:1 to per-CPU pvclock time info.
+	 * 2. that per-CPU pvclock time info is updated if the
+	 *    underlying CPU changes.
+	 * 3. that version is increased whenever underlying CPU
+	 *    changes.
+	 *
 	 */
 	do {
 		cpu = __getcpu() & VGETCPU_CPU_MASK;

@@ -104,8 +107,6 @@ static notrace cycle_t vread_pvclock(int *mode)
 
 		pvti = get_pvti(cpu);
 
-		migrate_count = pvti->migrate_count;
-
 		version = __pvclock_read_cycles(&pvti->pvti, &ret, &flags);
 
 		/*

@@ -117,8 +118,7 @@ static notrace cycle_t vread_pvclock(int *mode)
 		cpu1 = __getcpu() & VGETCPU_CPU_MASK;
 	} while (unlikely(cpu != cpu1 ||
 			  (pvti->pvti.version & 1) ||
-			  pvti->pvti.version != version ||
-			  pvti->migrate_count != migrate_count));
+			  pvti->pvti.version != version));
 
 	if (unlikely(!(flags & PVCLOCK_TSC_STABLE_BIT)))
 		*mode = VCLOCK_NONE;

include/linux/sched.h +0 -8

index 178a8d9..217ce4b 100644
@@ -107,14 +107,6 @@ extern unsigned long this_cpu_load(void);
 extern void calc_global_load(unsigned long ticks);
 extern void update_cpu_load_nohz(void);
 
-/* Notifier for when a task gets migrated to a new CPU */
-struct task_migration_notifier {
-	struct task_struct *task;
-	int from_cpu;
-	int to_cpu;
-};
-extern void register_task_migration_notifier(struct notifier_block *n);
-
 extern unsigned long get_parent_ip(unsigned long addr);
 
 extern void dump_cpu_task(int cpu);

kernel/sched/core.c +0 -9

index e8b3350..122e499 100644
@@ -974,13 +974,6 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)
 		rq->skip_clock_update = 1;
 }
 
-static ATOMIC_NOTIFIER_HEAD(task_migration_notifier);
-
-void register_task_migration_notifier(struct notifier_block *n)
-{
-	atomic_notifier_chain_register(&task_migration_notifier, n);
-}
-
 #ifdef CONFIG_SMP
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {

@@ -1011,18 +1004,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	trace_sched_migrate_task(p, new_cpu);
 
 	if (task_cpu(p) != new_cpu) {
-		struct task_migration_notifier tmn;
-
 		if (p->sched_class->migrate_task_rq)
 			p->sched_class->migrate_task_rq(p, new_cpu);
 		p->se.nr_migrations++;

kernel/sched/core.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 1013

Quoted 10 lines

kernel/sched/core.c +0 -6

 		perf_sw_event(PERF_COUNT_SW_CPU_MIGRATIONS, 1, NULL, 0);
-
-		tmn.task = p;
-		tmn.from_cpu = task_cpu(p);
-		tmn.to_cpu = new_cpu;
-
-		atomic_notifier_call_chain(&task_migration_notifier, 0, &tmn);
 	}
 
 	__set_task_cpu(p, new_cpu);

--
Gleb.

Peter Zijlstra

author maintainer

2013-08-23 10:22 UTC #8 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

In reply to Marcelo Tosatti

Quoted 11 lines · Marcelo Tosatti

Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.

Quoted 4 lines

Which is sufficient to guarantee nanoseconds counter is calculated properly.
Signed-off-by Marcelo Tosatti <[email protected]>

What happened to this patch? I don't see it in my current tree, nor do I find it in the kvm tree.

Gleb Natapov

2013-08-23 10:32 UTC #9 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

Quoted 12 lines · Peter Zijlstra

Quoted 11 lines · Marcelo Tosatti
Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.

Quoted 8 lines

Quoted 4 lines
Which is sufficient to guarantee nanoseconds counter is calculated properly.
Signed-off-by Marcelo Tosatti <[email protected]>
What happened to this patch? I don't see it in my current tree, nor do I find it in the kvm tree.

e04c5d76b0cfb66cadd900cf147526f2271884b8 in kvm.git next branch.

--
Gleb.

Peter Zijlstra

author maintainer

2013-08-23 10:35 UTC #10 lore raw

Re: [PATCH] remove sched notifier for cross-cpu migrations

Quoted 16 lines · Gleb Natapov

Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Marcelo Tosatti
Linux as a guest on KVM hypervisor, the only user of the pvclock vsyscall interface, does not require notification on task migration because:
1. cpu ID number maps 1:1 to per-CPU pvclock time info.
2. per-CPU pvclock time info is updated if the underlying CPU changes.
3. that version is increased whenever underlying CPU changes.
Which is sufficient to guarantee nanoseconds counter is calculated properly.

Quoted 6 lines

Quoted 5 lines
Quoted 1 line
Signed-off-by Marcelo Tosatti <[email protected]>
What happened to this patch? I don't see it in my current tree, nor do I find it in the kvm tree.
e04c5d76b0cfb66cadd900cf147526f2271884b8 in kvm.git next branch.

Awesome, thanks!