Skip to contentSkip to conversation

SDP perf drop with 2.6.15

DISCUSSION Grant Grundler 2 participants 3 messages Jan 12, 2006 – Jan 31, 2006

lore
Share & tools
lore
#0 lore raw
Quoted 4 lines · Michael S. Tsirkin
Quoted 2 lines

Makes me think it's a change in the interrupt code path. Any clues on where I might start with this?

oprofile should show where we are spending the time.

Executive summary:
Silly me. interrupt bindings changed and I didn't check after rolling to 2.6.15 (from 2.6.14). That explains the increase cpu utilization with lower performance.
Getting similar performance with -T 0,0 (2.6.15) vs -T 1,1 (2.6.14).

I can't explain why q-syscollect *improves* perf by ~11 to 17%. I cc'd linux-ia64 hoping someone might explain it.

Details:
Since I prefer q-syscollect:

grundler@gsyprf3:~/openib-perf-2006/rx2600-r4929$ LD_PRELOAD=/usr/local/lib/libsdp.so q-syscollect /usr/local/bin/netperf -p 12866 -l 60 -H 10.0.0.30 -t TCP_RR -T 1,1 -c -C -- -r 1,1 -s 0 -S 0
libsdp.so: $LIBSDP_CONFIG_FILE not set. Using /usr/local/etc/libsdp.conf
libsdp.so: $LIBSDP_CONFIG_FILE not set. Using /usr/local/etc/libsdp.conf
bind_to_specific_processor: enter
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.30 (10.0.0.30) port 0 AF_INET
Local /Remote
Socket Size   Request Resp.  Elapsed Trans.   CPU    CPU    S.dem   S.dem
Send   Recv   Size    Size   Time    Rate     local  remote local   remote
bytes  bytes  bytes   bytes  secs.   per sec  % S    % S    us/Tr   us/Tr

16384 87380 1 1 60.00 16300.11 10.53 8.26 12.925 10.137

Wierd. Performance jumps from 13900 to 16300 (+2400 or +%17). Hrm...something got me to look at /proc/interrupts and I see that mthca is interrupting on CPU0 now:

 70:  644084899          0       PCI-MSI-X  ib_mthca (comp)
 71:          8          0       PCI-MSI-X  ib_mthca (async)
 72:      27247          0       PCI-MSI-X  ib_mthca (cmd)

Retest with -T 0,1 :

16384 87380 1 1 60.00 17557.94 6.06 10.88 6.909 12.390

And again -T 0,1 but without q-syscollect:
16384 87380 1 1 60.00 15891.41 6.13 7.61 7.713 9.571

Now with -T 0,0:
16384 87380 1 1 60.00 20719.03 5.93 5.26 5.724 5.076

with -T 0,0 and without q-syscollect:
16384 87380 1 1 60.00 18553.61 5.73 5.36 6.181 5.782

That's +11% on the last set.
I'm stumped why q-syscollect would *improve* performance. Maybe someone on linux-ia64 has a theory?

.q/ output from the runs below is available on:
http://gsyprf3.external.hp.com/~grundler/openib-perf-2006/rx2600-r4929/.q

The "insteresting" output files from the last q-syscollect run:

	netperf-pid5693-cpu0.info#0
	q-syscollect-pid5691-cpu0.info#0
	unknown-cpu0.info#2

q-view output for last run of netperf/SDP is:
Command: /usr/local/bin/netperf -p 12866 -l 60 -H 10.0.0.30 -t TCP_RR -T 0 0 -c -C -- -r 1 1 -s 0 -S 0 Flat profile of CPU_CYCLES in netperf-pid5693-cpu0.hist#0:
Each histogram sample counts as 1.00034m seconds

 % time      self     cumul     calls self/call  tot/call name
  27.18      1.52      1.52     21.8M     69.9n     69.9n _spin_unlock_irqrestore<kernel>
  16.08      0.90      2.42     1.18M      761n      846n schedule<kernel>
   6.24      0.35      2.77     1.29M      271n     2.56u sdp_inet_recv<kernel>
   4.33      0.24      3.01     2.59M     93.6n     93.6n __kernel_syscall_via_break<kernel>
   3.90      0.22      3.23     1.18M      185n      342n sdp_send_buff_post<kernel>
   3.51      0.20      3.43     1.24M      158n      933n sdp_inet_send<kernel>
   2.70      0.15      3.58     2.48M     60.8n     60.8n kmem_cache_free<kernel>
   2.11      0.12      3.69     4.82M     24.5n     24.5n kmem_cache_alloc<kernel>
   1.93      0.11      3.80     1.20M     90.1n      358n sdp_recv_flush<kernel>
   1.79      0.10      3.90     2.53M     39.5n     39.5n fget<kernel>
   1.75      0.10      4.00     1.22M     80.2n     2.95u sys_recv<kernel>
   1.65      0.09      4.09     1.22M     75.4n      414n sdp_send_data_queue_test<kernel>
   1.59      0.09      4.18     1.25M     71.3n     1.16u sys_send<kernel>
   1.54      0.09      4.27         -         -         - send_tcp_rr
   1.41      0.08      4.35     2.33M     33.9n     33.9n sdp_buff_q_put_tail<kernel>
   1.22      0.07      4.41     21.7M     3.14n     3.14n _spin_lock_irqsave<kernel>
   1.09      0.06      4.48     1.25M     48.7n     48.7n __divsi3<kernel>
   1.06      0.06      4.53     1.23M     48.1n     2.86u sys_recvfrom<kernel>
   1.02      0.06      4.59     2.29M     24.9n     24.9n sba_map_single<kernel>
   1.02      0.06      4.65     2.49M     22.9n     63.0n sockfd_lookup<kernel>
   0.98      0.06      4.70         -         -         - sdp_exit<kernel>
   0.98      0.06      4.76     1.26M     43.7n      967n sock_sendmsg<kernel>
   0.97      0.05      4.81     2.38M     22.7n     72.3n sdp_buff_pool_get<kernel>
   0.95      0.05      4.87     2.50M     21.2n     21.2n fput<kernel>
   0.91      0.05      4.92     1.23M     41.4n     2.72u sock_recvmsg<kernel>
   0.89      0.05      4.97     1.19M     42.1n     62.9n memcpy_toiovec<kernel>
   0.88      0.05      5.02     2.50M     19.6n     19.6n __copy_user<kernel>
   0.86      0.05      5.06     1.21M     39.8n      868n schedule_timeout<kernel>
   0.82      0.05      5.11     1.29M     35.7n      128n send
   0.79      0.04      5.15     1.25M     35.1n     35.1n sched_clock<kernel>
   0.64      0.04      5.19     1.29M     27.8n      123n recv
   0.59      0.03      5.22     1.23M     26.9n     26.9n __udivdi3<kernel>
   0.59      0.03      5.26     1.19M     27.8n     99.1n mthca_tavor_post_send<kernel>

....

q-view output for the q-syscollect process:
Command: q-syscollect /usr/local/bin/netperf -p 12866 -l 60 -H 10.0.0.30 -t TCP_ RR -T 0,0 -c -C -- -r 1,1 -s 0 -S 0
Flat profile of CPU_CYCLES in q-syscollect-pid5691-cpu0.hist#0:
Each histogram sample counts as 1.00034m seconds

 % time      self     cumul     calls self/call  tot/call name
  0.00      0.00      0.00       162      0.00      0.00 _spin_lock_irqsave<kernel>
  0.00      0.00      0.00       149      0.00      0.00 _spin_unlock_irqrestore <kernel>
  0.00      0.00      0.00       115      0.00      0.00 lock_timer_base<kernel>
  0.00      0.00      0.00       102      0.00      0.00 schedule<kernel>
  0.00      0.00      0.00      41.0      0.00      0.00 kfree<kernel>

...

Only notable thing here is it calls "schedule" and triggers an additional ~160 interrupts (my guess).

unknown-cpu0.info#2 is probably interesting too:

Flat profile of CPU_CYCLES in unknown-cpu0.hist#2:
Each histogram sample counts as 1.00034m seconds

 % time      self     cumul     calls self/call  tot/call name
  91.59     53.37     53.37      931k     57.3u     57.3u default_idle
   4.53      2.64     56.00     10.8M      244n      244n _spin_unlock_irqrestore
   1.78      1.04     57.04     1.27M      815n      885n schedule
   0.75      0.43     57.48     1.38M      315n     2.30u mthca_eq_int
   0.49      0.28     57.76     1.38M      206n     2.57u handle_IRQ_event
   0.19      0.11     57.87         -         -         - cpu_idle
   0.14      0.08     57.96     1.35M     62.1n     2.41u mthca_tavor_msi_x_interrupt
   0.10      0.06     58.02     1.32M     45.4n     2.07u mthca_cq_completion

...

thanks,
grant

#1 lore raw

Quoting r. Grant Grundler <[email protected]>:

Quoted 6 lines

Subject: Re: SDP perf drop with 2.6.15

Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines

Makes me think it's a change in the interrupt code path. Any clues on where I might start with this?

Quoted 8 lines
Quoted 1 line

oprofile should show where we are spending the time.

Executive summary:
Silly me. interrupt bindings changed and I didn't check after rolling to 2.6.15 (from 2.6.14). That explains the increase cpu utilization with lower performance.
Getting similar performance with -T 0,0 (2.6.15) vs -T 1,1 (2.6.14).

OK, so we are left with the ref count now.
I'm seeing this myself sometimes, I have some other stuff to attend to next week, and then I plan to look into this.

Signature · 1 line
MST
#2 lore raw
In reply to Grant Grundler

On Thu, Jan 12, 2006 at 11:50:04AM -0800, Grant Grundler wrote:
...

Quoted 1 line

I can't explain why q-syscollect *improves* perf by ~11 to 17%.

Kudos to Stephane Eranian for sorting this out.

Execute summary:
Rebooting with "nohalt" kernel option gets me the full performance.

Gory details:
By default, ia64-linux goes to a "low power state" (no jokes about this please) in the idle loop. This is implemented with form of "halt" instruction. Perfmon subsystem disables use of "halt" since older (and possibly current) PAL support for "halt" was broken and it would break the Performance Monitoring HW state. Please ask Stephane offline if you need more details.

Stephane also commented that this might be an issue with other architectures as well if they have a low power state. The transition from low power to "normal" will have a cost on every architecture. And every interrupt is likely to incur that cost. This is a real problem for benchmarking where "latency" (ie we idle for very short periods of time) is hurt as I saw with netperf TCP_RR.

Q to ia64-linux: since perfmon can enable/disable this on the fly, can I add a /sys hook to do the same from userspace?
Where under /sys could this live?

cheers,
grant

Quoted 36 lines

Details:
Since I prefer q-syscollect:

grundler@gsyprf3:~/openib-perf-2006/rx2600-r4929$ LD_PRELOAD=/usr/local/lib/libsdp.so q-syscollect /usr/local/bin/netperf -p 12866 -l 60 -H 10.0.0.30 -t TCP_RR -T 1,1 -c -C -- -r 1,1 -s 0 -S 0
libsdp.so: $LIBSDP_CONFIG_FILE not set. Using /usr/local/etc/libsdp.conf
libsdp.so: $LIBSDP_CONFIG_FILE not set. Using /usr/local/etc/libsdp.conf
bind_to_specific_processor: enter
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.30 (10.0.0.30) port 0 AF_INET
Local /Remote
Socket Size   Request Resp.  Elapsed Trans.   CPU    CPU    S.dem   S.dem
Send   Recv   Size    Size   Time    Rate     local  remote local   remote
bytes  bytes  bytes   bytes  secs.   per sec  % S    % S    us/Tr   us/Tr

16384 87380 1 1 60.00 16300.11 10.53 8.26 12.925 10.137

Wierd. Performance jumps from 13900 to 16300 (+2400 or +%17). Hrm...something got me to look at /proc/interrupts and I see that mthca is interrupting on CPU0 now:

 70:  644084899          0       PCI-MSI-X  ib_mthca (comp)
 71:          8          0       PCI-MSI-X  ib_mthca (async)
 72:      27247          0       PCI-MSI-X  ib_mthca (cmd)

Retest with -T 0,1 :

16384 87380 1 1 60.00 17557.94 6.06 10.88 6.909 12.390

And again -T 0,1 but without q-syscollect:
16384 87380 1 1 60.00 15891.41 6.13 7.61 7.713 9.571

Now with -T 0,0:
16384 87380 1 1 60.00 20719.03 5.93 5.26 5.724 5.076

with -T 0,0 and without q-syscollect:
16384 87380 1 1 60.00 18553.61 5.73 5.36 6.181 5.782

That's +11% on the last set.
I'm stumped why q-syscollect would *improve* performance.

...