Skip to contentSkip to conversation

2.6.3-rc2 v 2.6.3-rc3-mm1 kernbench

BENCHMARK Con Kolivas 3 participants 8 messages Feb 16, 2004 – Feb 18, 2004

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here's some nice evidence of the sched domains' patch value:
kernbench 0.20 running on an X440 8x1.5Ghz P4HT (2 node)

Time is in seconds. Lower is better (fixed font table)

Summary:
Kernel: 2.6.3-rc2 2.6.3-rc3-mm1
Half(-j8) 120.8 113.0
Optimal(-j64) 81.6 79.3
Max(-j) 82.9 80.3

shorter summary:
2.6.3-rc3-mm1 kicks butt

long winded summary (look at the massive context switch differences):

results.2.6.3-rc2

Average Half Load Run:
Elapsed Time 120.808
User Time 802.428
System Time 92.072
Percent CPU 740
Context Switches 10613.6
Sleeps 26667

Average Optimum Load Run:
Elapsed Time 81.59
User Time 1007.89
System Time 112.36
Percent CPU 1372.6
Context Switches 63006.2
Sleeps 40406

Average Maximum Load Run:
Elapsed Time 82.944
User Time 1012.33
System Time 122.424
Percent CPU 1367.6
Context Switches 44822.2
Sleeps 22161

results.2.6.3-rc3-mm1:

Average Half Load Run:
Elapsed Time 113.008
User Time 742.786
System Time 90.65
Percent CPU 738
Context Switches 28062.6
Sleeps 24571.8

Average Optimum Load Run:
Elapsed Time 79.278
User Time 1007.69
System Time 107.388
Percent CPU 1407
Context Switches 33355
Sleeps 32720

Average Maximum Load Run:
Elapsed Time 80.33
User Time 1009.89
System Time 121.518
Percent CPU 1408.4
Context Switches 31802.4
Sleeps 22905

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAML6+ZUg7+tp6mRURAiH3AJ0bNPTLxNncxVoT1LivWhe4sXrAyQCeJzXw 6IsBVGzd4yJpR9eW3gZYBPM=
=tDlR
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 15 lines · Con Kolivas

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here's some nice evidence of the sched domains' patch value:
kernbench 0.20 running on an X440 8x1.5Ghz P4HT (2 node)

Time is in seconds. Lower is better (fixed font table)

Summary:
Kernel: 2.6.3-rc2 2.6.3-rc3-mm1
Half(-j8) 120.8 113.0
Optimal(-j64) 81.6 79.3
Max(-j) 82.9 80.3

Quoted 4 lines

shorter summary:
2.6.3-rc3-mm1 kicks butt

Thanks Con,
Results look pretty good. The half-load context switches are increased - that is probably a result of active balancing. And speaking of active balancing, it is not yet working across nodes with the configuration you're on.

To get some idea of our worst case SMT performance (-j8), would it be possible to do -j8 and -j64 runs with HT turned off?

#2 lore raw
Quoted 19 lines · Nick Piggin
Quoted 15 lines · Con Kolivas

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here's some nice evidence of the sched domains' patch value:
kernbench 0.20 running on an X440 8x1.5Ghz P4HT (2 node)

Time is in seconds. Lower is better (fixed font table)

Summary:
Kernel: 2.6.3-rc2 2.6.3-rc3-mm1
Half(-j8) 120.8 113.0
Optimal(-j64) 81.6 79.3
Max(-j) 82.9 80.3

Quoted 2 lines

shorter summary:
2.6.3-rc3-mm1 kicks butt

Quoted 8 lines

Thanks Con,
Results look pretty good. The half-load context switches are increased - that is probably a result of active balancing. And speaking of active balancing, it is not yet working across nodes with the configuration you're on.

To get some idea of our worst case SMT performance (-j8), would it be possible to do -j8 and -j64 runs with HT turned off?

sure.

results.2.6.3-rc3-mm1 + SMT:
Average Half Load Run:
Elapsed Time 113.008
User Time 742.786
System Time 90.65
Percent CPU 738
Context Switches 28062.6
Sleeps 24571.8

Average Optimum Load Run:
Elapsed Time 79.278
User Time 1007.69
System Time 107.388
Percent CPU 1407
Context Switches 33355
Sleeps 32720

2.6.3-rc3-mm1 no SMT:
Average Half Load Run:
Elapsed Time 133.51
User Time 799.268
System Time 92.784
Percent CPU 669
Context Switches 19340.8
Sleeps 24427.4

Average Optimum Load Run:
Elapsed Time 81.486
User Time 1006.37
System Time 106.952
Percent CPU 1366.8
Context Switches 33939
Sleeps 32453.4

Con

#3 lore raw

In article <[email protected]>, Con Kolivas <[email protected]> wrote:
| On Tue, 17 Feb 2004 00:20, Nick Piggin wrote:

| >
| > Thanks Con,
| > Results look pretty good. The half-load context switches are
| > increased - that is probably a result of active balancing.
| > And speaking of active balancing, it is not yet working across
| > nodes with the configuration you're on.
| >
| > To get some idea of our worst case SMT performance (-j8), would
| > it be possible to do -j8 and -j64 runs with HT turned off?
| 
| sure.

Now, I have a problem with the numbers here, either I don't understand them (likely) or I don't believe them (also possible).

| 
| results.2.6.3-rc3-mm1 + SMT:
| Average Half Load Run:
| Elapsed Time 113.008
| User Time 742.786
| System Time 90.65
| Percent CPU 738
| Context Switches 28062.6
| Sleeps 24571.8
| 2.6.3-rc3-mm1 no SMT:
| Average Half Load Run:
| Elapsed Time 133.51
| User Time 799.268
| System Time 92.784
| Percent CPU 669
| Context Switches 19340.8
| Sleeps 24427.4

As I look at these numbers, I see that with SMT the real time is lower, the system time is higher, and the system time is higher. All what I would expect since effectively the system has twice as many CPUs.

But the user time, there I have a problem understanding. User time is time in the user program, and I would expect user time to go up, since resource contention inside a CPU is likely to mean less work being done per unit of time, and therefore if you measure CPU time from the outside you need more of it to get the job done.

And what do you get running on one non-SMT CPU for the same mix? When I run stuff I usually see the user CPU go up a tad and the e.t. go down a little (SMT) or quite a bit (SMP).

I have faith in the reporting of the numbers, but I wonder about the way the data were measured. Hopefully someone can clarify, because it looks a little like what you would see if you counted "one" for one tick worth of user mode time in the CPU, regardless of whether one or two threads were executing.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#4 lore raw
Quoted 50 lines · bill davidsen

In article <[email protected]>, Con Kolivas <[email protected]> wrote:
| On Tue, 17 Feb 2004 00:20, Nick Piggin wrote:

| >
| > Thanks Con,
| > Results look pretty good. The half-load context switches are
| > increased - that is probably a result of active balancing.
| > And speaking of active balancing, it is not yet working across
| > nodes with the configuration you're on.
| >
| > To get some idea of our worst case SMT performance (-j8), would
| > it be possible to do -j8 and -j64 runs with HT turned off?
| | sure.

Now, I have a problem with the numbers here, either I don't understand them (likely) or I don't believe them (also possible).

| | results.2.6.3-rc3-mm1 + SMT:
| Average Half Load Run:
| Elapsed Time 113.008
| User Time 742.786
| System Time 90.65
| Percent CPU 738
| Context Switches 28062.6
| Sleeps 24571.8
| 2.6.3-rc3-mm1 no SMT:
| Average Half Load Run:
| Elapsed Time 133.51
| User Time 799.268
| System Time 92.784
| Percent CPU 669
| Context Switches 19340.8
| Sleeps 24427.4

As I look at these numbers, I see that with SMT the real time is lower, the system time is higher, and the system time is higher. All what I would expect since effectively the system has twice as many CPUs.

But the user time, there I have a problem understanding. User time is time in the user program, and I would expect user time to go up, since resource contention inside a CPU is likely to mean less work being done per unit of time, and therefore if you measure CPU time from the outside you need more of it to get the job done.

And what do you get running on one non-SMT CPU for the same mix? When I run stuff I usually see the user CPU go up a tad and the e.t. go down a little (SMT) or quite a bit (SMP).

Quoted 6 lines

I have faith in the reporting of the numbers, but I wonder about the way the data were measured. Hopefully someone can clarify, because it looks a little like what you would see if you counted "one" for one tick worth of user mode time in the CPU, regardless of whether one or two threads were executing.

Bill, I have CC'ed your message without modification because Con is not subscribed to the list. Even for people who are subscribed, the convention on lkml is to reply to all.

Anyway, the "no SMT" run is with CONFIG_SCHED_SMT turned off, P4 HT is still on. This was my fault because I didn't specify clearly that I wanted to see a run with hardware HT turned off, although these numbers are still interesting.

Con hasn't tried HT off AFAIK because we couldn't work out how to turn it off at boot time! :(

#5 lore raw
Quoted 11 lines · Nick Piggin

Bill, I have CC'ed your message without modification because Con is not subscribed to the list. Even for people who are subscribed, the convention on lkml is to reply to all.

Anyway, the "no SMT" run is with CONFIG_SCHED_SMT turned off, P4 HT is still on. This was my fault because I didn't specify clearly that I wanted to see a run with hardware HT turned off, although these numbers are still interesting.

Con hasn't tried HT off AFAIK because we couldn't work out how to turn it off at boot time! :(

The curse of the brain-dead BIOS :-(

So does CONFIG_SCHED_SMT turned off mean not using more than one sibling per package, or just going back to using them poorly? Yes, I should go root through the code.

Clearly it would be good to get one more data point with HT off in BIOS, but from this data it looks as if the SMT stuff really helps little when the system is very heavily loaded (Nproc>=Nsibs), and does best when the load is around Nproc==Ncpu. At least as I read the data. The really interesting data would be the -j64 load without HT, using both schedulers.

I just got done looking at a mail server with HT, kept the load avg 40-70 for a week. Speaks highly for the stability of RHEL-3.0, but I wouldn't mind a little more performance for free.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#6 lore raw
Quoted 17 lines · Bill Davidsen
Quoted 4 lines · Nick Piggin

Bill, I have CC'ed your message without modification because Con is not subscribed to the list. Even for people who are subscribed, the convention on lkml is to reply to all.

Quoted 8 lines

Anyway, the "no SMT" run is with CONFIG_SCHED_SMT turned off, P4 HT is still on. This was my fault because I didn't specify clearly that I wanted to see a run with hardware HT turned off, although these numbers are still interesting.

Con hasn't tried HT off AFAIK because we couldn't work out how to turn it off at boot time! :(

Quoted 7 lines

The curse of the brain-dead BIOS :-(

So does CONFIG_SCHED_SMT turned off mean not using more than one sibling per package, or just going back to using them poorly? Yes, I should go root through the code.

It just goes back to treating them the same as physical CPUs. The option will be eventually removed.

Quoted 7 lines

Clearly it would be good to get one more data point with HT off in BIOS, but from this data it looks as if the SMT stuff really helps little when the system is very heavily loaded (Nproc>=Nsibs), and does best when the load is around Nproc==Ncpu. At least as I read the data. The really interesting data would be the -j64 load without HT, using both schedulers.

The biggest problems with SMT happen when 1 < Nproc < Nsibs, because every two processes that end up on one physical CPU leaves one physical CPU idle, and the non HT scheduler can't detect or correct this.

At higher numbers of processes, you fill all virtual CPUs, so physical CPUs don't become idle. You can still be smarter about cache and migration costs though.

Quoted 5 lines

I just got done looking at a mail server with HT, kept the load avg 40-70 for a week. Speaks highly for the stability of RHEL-3.0, but I wouldn't mind a little more performance for free.

Not sure if they have any sort of HT aware scheduler or not. If they do it is probably a shared runqueues type which is much the same as sched domains in terms of functionality. I don't think it would help much here though.

#7 lore raw

Quoting Nick Piggin <[email protected]>:

Quoted 5 lines
Quoted 4 lines · Bill Davidsen
Quoted 2 lines · Nick Piggin

Con hasn't tried HT off AFAIK because we couldn't work out how to turn it off at boot time! :(

The curse of the brain-dead BIOS :-(

Err not quite; it's an X440 that is 10000 miles away so I cant really access the bios easily :P

Quoted 11 lines
Quoted 3 lines

So does CONFIG_SCHED_SMT turned off mean not using more than one sibling per package, or just going back to using them poorly? Yes, I should go root through the code.

It just goes back to treating them the same as physical CPUs. The option will be eventually removed.

Quoted 4 lines

Clearly it would be good to get one more data point with HT off in BIOS, but from this data it looks as if the SMT stuff really helps little when the system is very heavily loaded (Nproc>=Nsibs), and does best when the load is around Nproc==Ncpu. At least as I read the data.

Actually that machine is 8 packages, 16 logical cpus and kernbench half load by default is set to cpus / 2. The idea behind that load is to minimise wasted idle time so this is where good tuning should show the most bonus as it does.

The really

Quoted 11 lines
Quoted 1 line

interesting data would be the -j64 load without HT, using both schedulers.

The biggest problems with SMT happen when 1 < Nproc < Nsibs, because every two processes that end up on one physical CPU leaves one physical CPU idle, and the non HT scheduler can't detect or correct this.

At higher numbers of processes, you fill all virtual CPUs, so physical CPUs don't become idle. You can still be smarter about cache and migration costs though.

Quoted 8 lines
Quoted 3 lines

I just got done looking at a mail server with HT, kept the load avg 40-70 for a week. Speaks highly for the stability of RHEL-3.0, but I wouldn't mind a little more performance for free.

Not sure if they have any sort of HT aware scheduler or not. If they do it is probably a shared runqueues type which is much the same as sched domains in terms of functionality. I don't think it would help much here though.

I think any bonus on the optimal and max loads on kernbench is remarkable since it's usually easy to keep all cpus busy when the load is 4xnum_cpus or higher. The fact that sched_domains shows a decent percentage bonus even with these loads speaks volumes about the effectiveness of this patch.

Con