Skip to contentSkip to conversation

calibrate_migration_costs takes ages on s390

DISCUSSION Heiko Carstens 4 participants 13 messages Feb 13, 2006 – Feb 16, 2006 s390

lore
Share & tools
lore
#0 lore raw

The boot sequence on s390 sometimes takes ages and we spend a very long time (up to one or two minutes) in calibrate_migration_costs. The time spent there differs from boot to boot. Also the calculated costs differ a lot. I've seen differences by up to a factor of 15 (yes, factor not percent). Also I doubt that making these measurements make much sense on a completely virtualized architecture where you cannot tell how much cpu time you will get anyway.
Is there any workaround or fix available so we can avoid seeing this?

Thanks,
Heiko

#1 lore raw

From: Heiko Carstens <[email protected]>
Date: Mon, 13 Feb 2006 11:26:34 +0100

Quoted 8 lines

The boot sequence on s390 sometimes takes ages and we spend a very long time (up to one or two minutes) in calibrate_migration_costs. The time spent there differs from boot to boot. Also the calculated costs differ a lot. I've seen differences by up to a factor of 15 (yes, factor not percent). Also I doubt that making these measurements make much sense on a completely virtualized architecture where you cannot tell how much cpu time you will get anyway.
Is there any workaround or fix available so we can avoid seeing this?

Things are not as slow, but definitely slow on sparc64 too, and it's also due to the migration cost calculations.

It's also really bad that it's using vmalloc(), for one thing, because this thrashes the TLB (some of us have 64-entry software replaced TLBs) and also because you can make no guarentees about how well the backing physical pages will distribute into the L2 cache.

As a result, wildly different run-to-run results can be expected particularly for systems with 1-way or 2-way set assosciative L2 caches, which are common on sparc64. I don't know about s390.

I think the migration cost calculator is way overboard and needs to be toned down a little bit.

#2 lore raw
Quoted 7 lines · David S. Miller

Things are not as slow, but definitely slow on sparc64 too, and it's also due to the migration cost calculations.

It's also really bad that it's using vmalloc(), for one thing, because this thrashes the TLB (some of us have 64-entry software replaced TLBs) and also because you can make no guarentees about how well the backing physical pages will distribute into the L2 cache.

the TLB trashing is intended, to calculate the worst-case migration cost. If userspace is TLB-intensive, it will trash TLBs just as much.

Quoted 3 lines

As a result, wildly different run-to-run results can be expected particularly for systems with 1-way or 2-way set assosciative L2 caches, which are common on sparc64. I don't know about s390.

s390 is clearly a special-base, being a virtual platform. But the calibration should be improved to work better on sparc64.

Do things get better if you fill out include/asm-sparc64/system.h's sched_cacheflush() function, to flush the L2 cache? That should at least make the cache state more or less reproducable across runs.

Ingo

#3 lore raw

From: Ingo Molnar <[email protected]>
Date: Mon, 13 Feb 2006 11:54:21 +0100

Quoted 3 lines

Do things get better if you fill out include/asm-sparc64/system.h's sched_cacheflush() function, to flush the L2 cache? That should at least make the cache state more or less reproducable across runs.

Yes, I tried to implement that, and it makes the migration cost calculation take 4 to 5 times longer, so I'm leaving it unimplemented.

#4 lore raw
In reply to Heiko Carstens
Quoted 8 lines · Heiko Carstens

The boot sequence on s390 sometimes takes ages and we spend a very long time (up to one or two minutes) in calibrate_migration_costs. The time spent there differs from boot to boot. Also the calculated costs differ a lot. I've seen differences by up to a factor of 15 (yes, factor not percent). Also I doubt that making these measurements make much sense on a completely virtualized architecture where you cannot tell how much cpu time you will get anyway. Is there any workaround or fix available so we can avoid seeing this?

which is the precise kernel version used? We toned down calibration a bit recently.

The immediate workaround would be to use the migration_cost=0 boot parameter.

Generally, i agree that it makes sense to not calibrate at all on virtual platforms. Does the patch below help? It gives virtual platforms a way to provide a default migration cost and thus avoid the boot-time calibration altogether. (I have tested it on x86, it does the expected thing.) This needs to hit v2.6.16 too.

Ingo

---------
introduce the CONFIG_DEFAULT_MIGRATION_COST method for an architecture to set the scheduler migration costs. This turns off automatic detection of migration costs. Makes sense on virtual platforms, where migration costs are hard to measure accurately.

----

 arch/s390/Kconfig |    4 ++++
 kernel/sched.c    |   13 ++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)
arch/s390/Kconfig +4 -0
Index: linux-robust-list.q/arch/s390/Kconfig
===================================================================
@@ -80,6 +80,10 @@ config HOTPLUG_CPU
 	  can be controlled through /sys/devices/system/cpu/cpu#.
 	  Say N if you want to disable CPU hotplug.
 
+config DEFAULT_MIGRATION_COST
+	int
+	default "1000000"
+
 config MATHEMU
 	bool "IEEE FPU emulation"
 	depends on MARCH_G5
kernel/sched.c +12 -1
Index: linux-robust-list.q/kernel/sched.c
===================================================================
@@ -5159,7 +5159,18 @@ static void init_sched_build_groups(stru
 #define MAX_DOMAIN_DISTANCE 32
 
 static unsigned long long migration_cost[MAX_DOMAIN_DISTANCE] =
-		{ [ 0 ... MAX_DOMAIN_DISTANCE-1 ] = -1LL };
+		{ [ 0 ... MAX_DOMAIN_DISTANCE-1 ] =
+/*
+ * Architectures may override the migration cost and thus avoid
+ * boot-time calibration. Unit is nanoseconds. Mostly useful for
+ * virtualized hardware:
+ */
+#ifdef CONFIG_DEFAULT_MIGRATION_COST
+			CONFIG_DEFAULT_MIGRATION_COST
+#else
+			-1LL
+#endif
+};
 
 /*
  * Allow override of migration cost - in units of microseconds.
#5 lore raw
Quoted 11 lines
Quoted 8 lines

The boot sequence on s390 sometimes takes ages and we spend a very long time (up to one or two minutes) in calibrate_migration_costs. The time spent there differs from boot to boot. Also the calculated costs differ a lot. I've seen differences by up to a factor of 15 (yes, factor not percent). Also I doubt that making these measurements make much sense on a completely virtualized architecture where you cannot tell how much cpu time you will get anyway. Is there any workaround or fix available so we can avoid seeing this?

which is the precise kernel version used? We toned down calibration a bit recently.

2.6.16-rc3.

Quoted 8 lines

The immediate workaround would be to use the migration_cost=0 boot parameter.

Generally, i agree that it makes sense to not calibrate at all on virtual platforms. Does the patch below help? It gives virtual platforms a way to provide a default migration cost and thus avoid the boot-time calibration altogether. (I have tested it on x86, it does the expected thing.) This needs to hit v2.6.16 too.

Yes, calibrate_migration_costs is very fast now. But it turned out that this was just hiding the real problem: if we have CONFIG_PREEMPT disabled the kernel gets (sometimes) unbelievably slow.
I think this happened somewhere between rc1 and rc3. Maybe Hannes knows more exactly when this happened the first time, since I always run with CONFIG_PREEMPT enabled.

Heiko

#6 lore raw
In reply to Ingo Molnar
Quoted 12 lines · Ingo Molnar
Quoted 8 lines · Heiko Carstens

The boot sequence on s390 sometimes takes ages and we spend a very long time (up to one or two minutes) in calibrate_migration_costs. The time spent there differs from boot to boot. Also the calculated costs differ a lot. I've seen differences by up to a factor of 15 (yes, factor not percent). Also I doubt that making these measurements make much sense on a completely virtualized architecture where you cannot tell how much cpu time you will get anyway. Is there any workaround or fix available so we can avoid seeing this?

Quoted 2 lines

which is the precise kernel version used? We toned down calibration a bit recently.

We did a bit of testing, -rc2-git3 + the patch below was still ok.

[PATCH] s390: earlier initialization of cpu_possible_map 9733e2407ad2237867cb13c04e7d619397fa3090

#7 lore raw
Quoted 4 lines · Olaf Hering

We did a bit of testing, -rc2-git3 + the patch below was still ok.

[PATCH] s390: earlier initialization of cpu_possible_map 9733e2407ad2237867cb13c04e7d619397fa3090

I need to double check, but -git5 + that patch was reported to be slow.

#8 lore raw
Quoted 6 lines
Quoted 4 lines

We did a bit of testing, -rc2-git3 + the patch below was still ok.

[PATCH] s390: earlier initialization of cpu_possible_map 9733e2407ad2237867cb13c04e7d619397fa3090

I need to double check, but -git5 + that patch was reported to be slow.

I did a quick git bisect search. This is one is the hurting one:

Author: Ingo Molnar <[email protected]> 2006-02-07 21:58:54
Committer: Linus Torvalds <[email protected]> 2006-02-08 01:12:33
Parent: 8519fb30e438f8088b71a94a7d5a660a814d3872 ([PATCH] mm: compound release fix)
Child: 0d4c3e7a8c65892c7d6a748fdbb4499e988880db ([PATCH] unshare system call -v5: Documentation file)

[PATCH] Fix spinlock debugging delays to not time out too early

The spinlock-debug wait-loop was using loops_per_jiffy to detect too long spinlock waits - but on fast CPUs this led to a way too fast timeout and false messages.

The fix is to include a __delay(1) call in the loop, to correctly approximate the intended delay timeout of 1 second. The code assumes that every architecture implements __delay(1) to last around 1/(loops_per_jiffy*HZ) seconds.

I guess we're once again suffering from being a virtualized platform: the formerly used call to cpu_relax() informed the underlying hypervisor that we want to give up the current cpu while __delay() keeps it. Unless we're scheduled away involuntarily.
The "Detect Soft Lockups" option doesn't make too much sense too on our platform, since we get a lot of false positives.
Quick fix: turn off the options CONFIG_DEBUG_SPINLOCK and CONFIG_DETECT_SOFTLOCKUP.

Heiko

#9 lore raw
Quoted 12 lines

I did a quick git bisect search. This is one is the hurting one:

Author: Ingo Molnar <[email protected]> 2006-02-07 21:58:54
Committer: Linus Torvalds <[email protected]> 2006-02-08 01:12:33
Parent: 8519fb30e438f8088b71a94a7d5a660a814d3872 ([PATCH] mm: compound release fix)
Child: 0d4c3e7a8c65892c7d6a748fdbb4499e988880db ([PATCH] unshare system call -v5: Documentation file)

The fix is to include a __delay(1) call in the loop, to correctly approximate the intended delay timeout of 1 second. The code assumes that every architecture implements __delay(1) to last around 1/(loops_per_jiffy*HZ) seconds.

Quoted 8 lines

I guess we're once again suffering from being a virtualized platform: the formerly used call to cpu_relax() informed the underlying hypervisor that we want to give up the current cpu while __delay() keeps it. Unless we're scheduled away involuntarily.
The "Detect Soft Lockups" option doesn't make too much sense too on our platform, since we get a lot of false positives.
Quick fix: turn off the options CONFIG_DEBUG_SPINLOCK and CONFIG_DETECT_SOFTLOCKUP.

Wrong analysis. Our __delay() implementation is broken. This doesn't help for the CONFIG_DETECT_SOFTLOCKUP case, but at least CONFIG_DEBUG_SPINLOCK works again with this.

Andrew, could you pick this one up, or should I send it separately?

[PATCH] s390: fix __delay implementation

From: Heiko Carstens <[email protected]>

Fix __delay implementation. Called with an argument "1" or "0" it would loop nearly forever (since (1/2)-1 = 0xffffffff).

---

 arch/s390/lib/delay.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/s390/lib/delay.c +1 -1
index e96c35b..71f0a2f 100644
@@ -30,7 +30,7 @@ void __delay(unsigned long loops)
          */
         __asm__ __volatile__(
                 "0: brct %0,0b"
-                : /* no outputs */ : "r" (loops/2) );
+                : /* no outputs */ : "r" ((loops/2) + 1));
 }
 
 /*
#10 lore raw

arch/s390/lib/delay.c void __delay(unsigned long loops) · line 34

Quoted 9 lines · Heiko Carstens
arch/s390/lib/delay.c +1 -1
@@ -30,7 +30,7 @@ void __delay(unsigned long loops)
          */
         __asm__ __volatile__(
                 "0: brct %0,0b"
-                : /* no outputs */ : "r" (loops/2) );
+                : /* no outputs */ : "r" ((loops/2) + 1));
 }

ahh ... that explains the delays indeed!

Ingo

#11 lore raw
Quoted 13 lines · Ingo Molnar

arch/s390/lib/delay.c void __delay(unsigned long loops) · line 34

Quoted 9 lines · Heiko Carstens
arch/s390/lib/delay.c +1 -1
@@ -30,7 +30,7 @@ void __delay(unsigned long loops)
          */
         __asm__ __volatile__(
                 "0: brct %0,0b"
-                : /* no outputs */ : "r" (loops/2) );
+                : /* no outputs */ : "r" ((loops/2) + 1));
 }

ahh ... that explains the delays indeed!

just to make sure, i've checked all the other __delay() implementations in the kernel, and none seems to have such problems.

Ingo

#12 lore raw
In reply to Ingo Molnar
Quoted 43 lines

introduce the CONFIG_DEFAULT_MIGRATION_COST method for an architecture to set the scheduler migration costs. This turns off automatic detection of migration costs. Makes sense on virtual platforms, where migration costs are hard to measure accurately.

----

 arch/s390/Kconfig |    4 ++++
 kernel/sched.c    |   13 ++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

Index: linux-robust-list.q/arch/s390/Kconfig
===================================================================

arch/s390/Kconfig +4 -0
@@ -80,6 +80,10 @@ config HOTPLUG_CPU
 	  can be controlled through /sys/devices/system/cpu/cpu#.
 	  Say N if you want to disable CPU hotplug.
 
+config DEFAULT_MIGRATION_COST
+	int
+	default "1000000"
+
 config MATHEMU
 	bool "IEEE FPU emulation"
 	depends on MARCH_G5

Index: linux-robust-list.q/kernel/sched.c
===================================================================

kernel/sched.c +6 -1
@@ -5159,7 +5159,18 @@ static void init_sched_build_groups(stru
 #define MAX_DOMAIN_DISTANCE 32
 
 static unsigned long long migration_cost[MAX_DOMAIN_DISTANCE] =
-		{ [ 0 ... MAX_DOMAIN_DISTANCE-1 ] = -1LL };
+		{ [ 0 ... MAX_DOMAIN_DISTANCE-1 ] =
+/*
+ * Architectures may override the migration cost and thus avoid
+ * boot-time calibration. Unit is nanoseconds. Mostly useful for
+ * virtualized hardware:
+ */

kernel/sched.c static void init_sched_build_groups(stru

Quoted 10 lines
kernel/sched.c +6 -1
+#ifdef CONFIG_DEFAULT_MIGRATION_COST
+			CONFIG_DEFAULT_MIGRATION_COST
+#else
+			-1LL
+#endif
+};
 
 /*
  * Allow override of migration cost - in units of microseconds.
-

This one should be applied then.

Thanks,
Heiko