2.5.38-mm2 dbench $N times
Taken on 32x/32G NUMA-Q:
Throughput 67.3949 MB/sec (NB=84.2436 MB/sec 673.949 MBit/sec) 16 procs dbench 16 11.72s user 122.21s system 422% cpu 31.733 total
Throughput 95.1519 MB/sec (NB=118.94 MB/sec 951.519 MBit/sec) 32 procs dbench 32 24.71s user 357.97s system 847% cpu 45.175 total
Throughput 93.8379 MB/sec (NB=117.297 MB/sec 938.379 MBit/sec) 64 procs dbench 64 56.03s user 773.39s system 903% cpu 1:31.75 total
Throughput 87.2713 MB/sec (NB=109.089 MB/sec 872.713 MBit/sec) 128 procs dbench 128 116.31s user 1524.85s system 840% cpu 3:15.16 total
Throughput 84.454 MB/sec (NB=105.567 MB/sec 844.54 MBit/sec) 192 procs dbench 192 180.64s user 2293.04s system 821% cpu 5:01.13 total
Throughput 82.9662 MB/sec (NB=103.708 MB/sec 829.662 MBit/sec) 224 procs dbench 224 212.30s user 2716.77s system 820% cpu 5:57.15 total
Throughput 37.9382 MB/sec (NB=47.4227 MB/sec 379.382 MBit/sec) 256 procs dbench 256 237.38s user 3115.41s system 376% cpu 14:51.40 total
Throughput 25.7546 MB/sec (NB=32.1932 MB/sec 257.546 MBit/sec) 512 procs dbench 512 465.96s user 5980.49s system 245% cpu 43:45.79 total
Cheers,
Bill
Quoted 6 lines · William Lee Irwin III
Taken on 32x/32G NUMA-Q:
Throughput 67.3949 MB/sec (NB=84.2436 MB/sec 673.949 MBit/sec) 16 procs dbench 16 11.72s user 122.21s system 422% cpu 31.733 total
Taken on 2x/0.8G el-scruffo PC:
Throughput 135.02 MB/sec (NB=168.775 MB/sec 1350.2 MBit/sec) ./dbench 16 12.11s user 16.29s system 181% cpu 15.646 total
What's up with that?
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
Taken on 32x/32G NUMA-Q:
Throughput 67.3949 MB/sec (NB=84.2436 MB/sec 673.949 MBit/sec) 16 procs dbench 16 11.72s user 122.21s system 422% cpu 31.733 total
Quoted 4 lines · Andrew Morton
Taken on 2x/0.8G el-scruffo PC:
Throughput 135.02 MB/sec (NB=168.775 MB/sec 1350.2 MBit/sec) ./dbench 16 12.11s user 16.29s system 181% cpu 15.646 total What's up with that?
Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
Cheers,
Bill
Quoted 4 lines · Andrew Morton
Quoted 4 lines
Taken on 2x/0.8G el-scruffo PC:
Throughput 135.02 MB/sec (NB=168.775 MB/sec 1350.2 MBit/sec) ./dbench 16 12.11s user 16.29s system 181% cpu 15.646 total What's up with that?
Quoted 3 lines · William Lee Irwin III
Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
Exact numbers:
Total throughput: 136.09139999999999 MB/s
dbench.log.j:Throughput 17.4581 MB/sec (NB=21.8226 MB/sec 174.581 MBit/sec) 64 procs dbench.log.k:Throughput 17.2604 MB/sec (NB=21.5755 MB/sec 172.604 MBit/sec) 64 procs dbench.log.l:Throughput 19.0192 MB/sec (NB=23.774 MB/sec 190.192 MBit/sec) 64 procs dbench.log.m:Throughput 15.7826 MB/sec (NB=19.7283 MB/sec 157.826 MBit/sec) 64 procs dbench.log.n:Throughput 15.8795 MB/sec (NB=19.8494 MB/sec 158.795 MBit/sec) 64 procs dbench.log.o:Throughput 17.621 MB/sec (NB=22.0263 MB/sec 176.21 MBit/sec) 64 procs dbench.log.p:Throughput 15.489 MB/sec (NB=19.3613 MB/sec 154.89 MBit/sec) 64 procs dbench.log.q:Throughput 17.5816 MB/sec (NB=21.977 MB/sec 175.816 MBit/sec) 64 procs
Quoted 7 lines · William Lee Irwin III
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
Taken on 32x/32G NUMA-Q:
Throughput 67.3949 MB/sec (NB=84.2436 MB/sec 673.949 MBit/sec) 16 procs dbench 16 11.72s user 122.21s system 422% cpu 31.733 totalOn Tue, Sep 24, 2002 at 10:47:30AM -0700, Andrew Morton wrote:
Quoted 9 lines
Quoted 4 lines
Taken on 2x/0.8G el-scruffo PC:
Throughput 135.02 MB/sec (NB=168.775 MB/sec 1350.2 MBit/sec) ./dbench 16 12.11s user 16.29s system 181% cpu 15.646 total What's up with that?Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Anton can get 3000 MByte/sec ;)
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
Quoted 4 lines · Andrew Morton
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Anton can get 3000 MByte/sec ;)
Hmm, this is odd (remember, dcache_rcu is in here):
c01053dc 9194801 60.668 poll_idle c01175db 1752528 11.5633 .text.lock.sched c0114c08 1281763 8.45717 load_balance c0106408 388517 2.56346 .text.lock.semaphore c0147a4e 272571 1.79844 .text.lock.file_table c0115080 265006 1.74853 scheduler_tick c0132374 227403 1.50042 generic_file_write_nolock c0115434 187759 1.23885 do_schedule c01233c8 167905 1.10785 run_timer_tasklet c0114778 103077 0.68011 try_to_wake_up c010603c 100121 0.660606 __down c0111788 96062 0.633824 smp_apic_timer_interrupt c011fa20 70840 0.467408 tasklet_hi_action c011f700 63125 0.416504 do_softirq c0131880 60640 0.400107 file_read_actor c0145dd0 53872 0.355452 generic_file_llseek c01473e0 45819 0.302317 get_empty_filp c0175190 35814 0.236304 ext2_new_block c01476ac 31657 0.208875 __fput c0146354 26755 0.176532 vfs_write c010d718 26627 0.175687 timer_interrupt c01a2cd0 26426 0.174361 atomic_dec_and_lock c0123294 23022 0.151901 update_one_process
Quoted 6 lines · William Lee Irwin III
...
c01053dc 9194801 60.668 poll_idle c01175db 1752528 11.5633 .text.lock.sched c0114c08 1281763 8.45717 load_balance c0106408 388517 2.56346 .text.lock.semaphore
lock_super in the ext2 block allocator, I bet.
Al was making noises about nailing that. Per-blockgroup rwlocks would be neat.
Quoted 7 lines · Andrew Morton
Quoted 5 lines · William Lee Irwin III
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
Taken on 32x/32G NUMA-Q:
Throughput 67.3949 MB/sec (NB=84.2436 MB/sec 673.949 MBit/sec) 16 procs dbench 16 11.72s user 122.21s system 422% cpu 31.733 total
Quoted 5 lines
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Anton can get 3000 MByte/sec ;)
Bill's Machine cost around $50, plus the cost to repair the walls that I crushed when hauling the pieces around. Anton's cost $2 million. Bill wins :)
Are you trying to bind the processes anywhere? I wonder what would happen if you make it always run quad 0...
Signature · 2 lines
Dave Hansen [email protected]
Quoted 4 lines · Andrew Morton
Quoted 4 lines
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Anton can get 3000 MByte/sec ;)
Quoted 5 lines · Dave Hansen
Bill's Machine cost around $50, plus the cost to repair the walls that I crushed when hauling the pieces around. Anton's cost $2 million. Bill wins :)
Are you trying to bind the processes anywhere? I wonder what would happen if you make it always run quad 0...
It's probably more an artifact of not having substantial I/O subsystems. This is basically a single-JBOD test.
Cheers,
Bill
Quoted 7 lines
Quoted 3 lines
Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Does dbench have any sort of CPU locality between who read it into pagecache, and who read it out again? If not, you stand 7/8 chance of being on the wrong node, and getting 1/20 of the mem bandwidth ....
M.
Quoted 4 lines · Martin J. Bligh
Quoted 4 lines
Quoted 3 lines
Not sure. This is boot bay SCSI crud, but single-disk FC looks *worse* for no obvious reason. Multiple disk tests do much better (about matching the el-scruffo PC numbers above).
Quoted 8 lines
Quoted 3 lines
dbench 16 on that sort of machine is a memory bandwidth test. And a dcache lock exerciser. It basically doesn't touch the disk. Something very bad is happening.
Does dbench have any sort of CPU locality between who read it into pagecache, and who read it out again? If not, you stand 7/8 chance of being on the wrong node, and getting 1/20 of the mem bandwidth ....
Pretty sure each dbench child does it's own write/read to only it's own data.
There is no sharing that I am aware of between the processes. How about
running in tmpfs to avoid any disk IO at all?
Also, what's the policy for home node assignment on fork? Are all of these children getting the same home node assingnment??
-Andrew
Quoted 2 lines
Pretty sure each dbench child does it's own write/read to only it's own data. There is no sharing that I am aware of between the processes.
Right, but if the processes migrate easily, there's still no CPU locality. Bill, do you want to try binding 1/32 of the processes to each CPU, and see if that makes your throughput increase?
Quoted 1 line
How about running in tmpfs to avoid any disk IO at all?
As far as I understand it, that won't help - we're operating out of pagecache anyway at this level, I think.
Quoted 2 lines
Also, what's the policy for home node assignment on fork? Are all of these children getting the same home node assingnment??
Policy is random - the scheduler lacks any NUMA comprehension at the moment. The numa sched mods would probably help a lot too.
M.
Quoted 6 lines · Martin J. Bligh
Quoted 2 lines
Pretty sure each dbench child does it's own write/read to only it's own data. There is no sharing that I am aware of between the processes.
Right, but if the processes migrate easily, there's still no CPU locality. Bill, do you want to try binding 1/32 of the processes to each CPU, and see if that makes your throughput increase?
Any stats to track migration over <x> time? Would be intersting to see this in some sort of /proc/pid maybe?
Quoted 4 lines
Quoted 1 line
How about running in tmpfs to avoid any disk IO at all?
As far as I understand it, that won't help - we're operating out of pagecache anyway at this level, I think.
at dbench 512, that's about 512*20MB, 10GB (not 100% sure on that 20MB figure). Have we hit a threshold at which this stuff gets written to disk?
Quoted 5 lines
Quoted 2 lines
Also, what's the policy for home node assignment on fork? Are all of these children getting the same home node assingnment??
Policy is random - the scheduler lacks any NUMA comprehension at the moment. The numa sched mods would probably help a lot too.
Oops, confusing this run with the numa sched stuff earlier.
Andrew Theurer
Quoted 4 lines · Martin J. Bligh
Quoted 4 lines
Does dbench have any sort of CPU locality between who read it into pagecache, and who read it out again? If not, you stand 7/8 chance of being on the wrong node, and getting 1/20 of the mem bandwidth ....
Quoted 3 lines · Andrew Theurer
Pretty sure each dbench child does it's own write/read to only it's own data. There is no sharing that I am aware of between the processes. How about running in tmpfs to avoid any disk IO at all?
tmpfs needs some fixes before it can be used for that. Hugh's working on it.
Cheers,
Bill