IO CPU affinity update:

Hi Jens -

Two patches:

1. Adds in the IRQ saving to generic_smp_call_function_single_interrupt (as you had suggested).
2. Ensures a single IPI generated to get a remote function call handler going.

So far it is working better than before on the 4-way IA64 w/ the mkfs/untar/make test suite - after 22 runs:

Part RQ MIN AVG MAX Dev

----- --  ------  ------  ------  ------
 mkfs  0  18.786  19.253  19.655   0.241
 mkfs  1  18.639  19.182  19.786   0.293

untar  0  17.140  17.486  18.250   0.322
untar  1  16.951  17.494  18.274   0.350

 make  0  22.927  24.310  34.339   2.287
 make  1  22.863  23.788  24.189   0.333

 comb  0  59.478  61.049  70.320   2.142
 comb  1  59.875  60.463  61.305   0.458

 psys  0   3.96%   4.14%   4.39%   0.100
 psys  1   3.60%   3.85%   4.19%   0.176

So we're seeing reduced time (~1.0%) and reduced %sys to do it (7.0%). The tighter deviations for make with rq=1 may be interesting... :-)

I've compiled & booted the patches for x86_64 - rq=1 is working on that platform too.

Quoted 29 lines · Alan D. Brunelle

Hi Jens -

Two patches:

1. Adds in the IRQ saving to generic_smp_call_function_single_interrupt (as you had suggested).
2. Ensures a single IPI generated to get a remote function call handler going.

So far it is working better than before on the 4-way IA64 w/ the mkfs/untar/make test suite - after 22 runs:

Part RQ MIN AVG MAX Dev

----- --  ------  ------  ------  ------
 mkfs  0  18.786  19.253  19.655   0.241
 mkfs  1  18.639  19.182  19.786   0.293

untar  0  17.140  17.486  18.250   0.322
untar  1  16.951  17.494  18.274   0.350

 make  0  22.927  24.310  34.339   2.287
 make  1  22.863  23.788  24.189   0.333

 comb  0  59.478  61.049  70.320   2.142
 comb  1  59.875  60.463  61.305   0.458

 psys  0   3.96%   4.14%   4.39%   0.100
 psys  1   3.60%   3.85%   4.19%   0.176

So we're seeing reduced time (~1.0%) and reduced %sys to do it (7.0%). The tighter deviations for make with rq=1 may be interesting... :-)

Quoted 2 lines

I've compiled & booted the patches for x86_64 - rq=1 is working on that platform too.

This is starting to look pretty good! Thanks a lot for these results, and the ->activated optimizations. I had a feeling the unstable results were something like this, missing ipi's.

Quoted 30 lines · Jens Axboe

Quoted 29 lines · Alan D. Brunelle

Hi Jens -

Two patches:

1. Adds in the IRQ saving to generic_smp_call_function_single_interrupt (as you had suggested).
2. Ensures a single IPI generated to get a remote function call handler going.

So far it is working better than before on the 4-way IA64 w/ the mkfs/untar/make test suite - after 22 runs:

Part RQ MIN AVG MAX Dev

----- --  ------  ------  ------  ------
 mkfs  0  18.786  19.253  19.655   0.241
 mkfs  1  18.639  19.182  19.786   0.293

untar  0  17.140  17.486  18.250   0.322
untar  1  16.951  17.494  18.274   0.350

 make  0  22.927  24.310  34.339   2.287
 make  1  22.863  23.788  24.189   0.333

 comb  0  59.478  61.049  70.320   2.142
 comb  1  59.875  60.463  61.305   0.458

 psys  0   3.96%   4.14%   4.39%   0.100
 psys  1   3.60%   3.85%   4.19%   0.176

So we're seeing reduced time (~1.0%) and reduced %sys to do it (7.0%). The tighter deviations for make with rq=1 may be interesting... :-)

Quoted 7 lines

Quoted 2 lines

I've compiled & booted the patches for x86_64 - rq=1 is working on that platform too.

This is starting to look pretty good! Thanks a lot for these results, and the ->activated optimizations. I had a feeling the unstable results were something like this, missing ipi's.

Jens: FYI: I am still seeing infrequent hangs on the x86_64-based platform. It happened again today after my patch was added, I did get <alt><sysrq><W> to work this time, and the threads that were stuck were waiting for IOs to complete. I believe at some point you were thinking of hacking in a block IO dump magic key as well - is that there yet?

I'll get to the ia64 testing tomorrow - have to run now, spent most of the day looking at what could cause stuff to be missed on x86_64. The code looks solid to me, but this hang needs to be figured out.

Lastly, I did do a run on a 16-way ia64 (with my patches) and again it ran fine.