direct IO submission and completion scalability issues

We have been looking into the linux kernel direct IO scalability issues with database workloads. Comments and suggestions on our below experiments are welcome.

In the linux kernel, direct IO requests are not batched at the block layer. i.e, as a new request comes in, the request get directly submitted to the IO controller on the same cpu that the request originates. And the IO completion likely happens on a different cpu which is processing interrupts. This results in cacheline bouncing of some of the hot kernel cachelines (like timers, scsi cmds, slab, sched, etc) and is becoming an important scalability issue as the number of cpus and distance between them increase with multi-core and numa.

Second experiment which we did was migrating the IO submission to the IO completion cpu. Instead of submitting the IO on the same cpu where the request arrived, in this experiment the IO submission gets migrated to the cpu that is processing IO completions(interrupt). This will minimize the access to remote cachelines (that happens in timers, slab, scsi layers). The IO submission request is forwarded to the kblockd thread on the cpu receiving the interrupts. As part of this, we also made kblockd thread on each cpu as the highest priority thread, so that IO gets submitted as soon as possible on the interrupt cpu with out any delay. On x86_64 SMP platform with 16 cores, this resulted in 2% performance improvement and 3.3% improvement on two node ia64 platform.

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

Quoted 3 lines · Suresh B

We have been looking into the linux kernel direct IO scalability issues with database workloads. Comments and suggestions on our below experiments are welcome.

This was on an SMP system? These issues are much more pronounced on a NUMA system. There the locality of the device may be a prime issue.

Yes. The issue is even worse if the submission comes from a remote node. F.e. If we have a system with a scsi controller on node 2. Now I/O submission on node 1 and completion on node 2. In that case the cacheline has to be transferred across the NUMA interlink.

However, you cannot avoid running the completion on the node where the device sits. The device has all sorts of control structures and if you would handle the completion on node 1 then it would have to transfer lots of cachelines that contain device state to node 1.

I think it is better to leave things as is. Or have the I/O submission be relocated to the node of the device.

Quoted 11 lines

Second experiment which we did was migrating the IO submission to the IO completion cpu. Instead of submitting the IO on the same cpu where the request arrived, in this experiment the IO submission gets migrated to the cpu that is processing IO completions(interrupt). This will minimize the access to remote cachelines (that happens in timers, slab, scsi layers). The IO submission request is forwarded to the kblockd thread on the cpu receiving the interrupts. As part of this, we also made kblockd thread on each cpu as the highest priority thread, so that IO gets submitted as soon as possible on the interrupt cpu with out any delay. On x86_64 SMP platform with 16 cores, this resulted in 2% performance improvement and 3.3% improvement on two node ia64 platform.

I think that is the right approach. This will also help in cases where I/O devices can only be accessed from a certain node (NUMA device address restrictions on some systems may not allow remote cacheline access!)

Quoted 7 lines

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

Right.

Quoted 10 lines · Christoph Lameter

Quoted 7 lines · Suresh B

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

Quoted 1 line

Right.

So any suggestions for making this clean and acceptable to everyone?

From then on, the submission is driven by completion, right?

Quoted 1 line · Suresh B

So any suggestions for making this clean and acceptable to everyone?

It is obviously a good idea to hand over the IO at the point which requires the least number of cachelines to be moved, and I think doing it in the block layer is right. Mostly you have to convince the block and driver maintainers I guess.

The scheduler really should be made interrupt-load aware anyway, so I don't have a problem with changing that; or scheduling kblockd at a higher priority, but I don't know if SCHED_FIFO is a good idea. Couldn't it be done in a softirq instead?

Latency for IO migration could be the most difficult problem to solve really. You don't give much details of the workload, profiles, etc... I hope this is for a real world test?

Can the locking be improved in simpler ways first?

Just some random questions...

It looks like the main source of cacheline bouncing you're eliminating is from the initial starting of IO from an empty queue (ie. unplug). From then on, the submission is driven by completion, right?

Why is the queue allowed to go empty in the first place in an IO critical workload?

Are you loading up each CPU with as many disks as it can possibly handle plus a few more? If so, is that realistic? (I honestly don't know).

You say that you'd like to do this for direct IO only, but if it is more efficient, why not for buffered IO as well? (or is it not more efficient for buffered IO? if not, why?)

AFAIKS, you'd still have significant queue_lock contention from other CPUs inserting requests into the list?

What IO scheduler are you using? I assume noop...

as a crazy experiment, what happens if you create per-cpu request queues?

Quoted 7 lines · Nick Piggin

Quoted 1 line · Suresh B

So any suggestions for making this clean and acceptable to everyone?

It is obviously a good idea to hand over the IO at the point which requires the least number of cachelines to be moved, and I think doing it in the block layer is right. Mostly you have to convince the block and driver maintainers I guess.

Yes. Implementation is the challenging part I guess.

Quoted 4 lines

The scheduler really should be made interrupt-load aware anyway, so I don't have a problem with changing that; or scheduling kblockd at a higher priority, but I don't know if SCHED_FIFO is a good idea. Couldn't it be done in a softirq instead?

Yes, softirq context is one way. But just didn't want to penalize the running task by taking away some of its cpu time. With CFS micro accounting, perhaps we can track irq, softirq time and avoid penalizing the running task's cpu time.

Quoted 3 lines

Latency for IO migration could be the most difficult problem to solve really. You don't give much details of the workload, profiles, etc... I hope this is for a real world test?

Improvement numbers quoted are from the OLTP database workload. We can look into other workloads.

Quoted 8 lines

Can the locking be improved in simpler ways first?

Just some random questions...

It looks like the main source of cacheline bouncing you're eliminating is from the initial starting of IO from an empty queue (ie. unplug). From then on, the submission is driven by completion, right?

Quoted 2 lines

Why is the queue allowed to go empty in the first place in an IO critical workload?

This workload is using direct IO and there is no batching at the block layer for direct IO. IO is submitted to the HW as it arrives.

Quoted 2 lines

Are you loading up each CPU with as many disks as it can possibly handle plus a few more? If so, is that realistic? (I honestly don't know).

There is 3-4% iowait time in the system. So the cpu's are not 100% busy, but there is quite a bit of direct IO going on.

Quoted 3 lines

You say that you'd like to do this for direct IO only, but if it is more efficient, why not for buffered IO as well? (or is it not more efficient for buffered IO? if not, why?)

It is applicable for both direct IO and buffered IO. But the implementations will differ. For example in buffered IO, we can setup in such a way that the block plug timeout function runs on the IO completion cpu.

Quoted 2 lines

AFAIKS, you'd still have significant queue_lock contention from other CPUs inserting requests into the list?

Correct. We have more potential to explore. Current implementation is very elementary.

Quoted 1 line

What IO scheduler are you using? I assume noop...

yes.

Quoted 1 line

as a crazy experiment, what happens if you create per-cpu request queues?

or in other words, each kblockd thread catering multiple request queues (perhaps one for each cpu or one for group of cpu's).

softirq context and each kblockd thread handling multiple request queues will lead to further improvements.

Quoted 4 lines · Suresh B

Yes, softirq context is one way. But just didn't want to penalize the running task by taking away some of its cpu time. With CFS micro accounting, perhaps we can track irq, softirq time and avoid penalizing the running task's cpu time.

But you "penalize" the running task in the completion handler as well anyway.

Doing this with a SCHED_FIFO task is sort of like doing interrupt threading which AFAIK has not been accepted (yet).

Quoted 2 lines

This workload is using direct IO and there is no batching at the block layer for direct IO. IO is submitted to the HW as it arrives.

So you aren't putting concurrent requests into the queue? Sounds like userspace should be improved.

Quoted 3 lines

It is applicable for both direct IO and buffered IO. But the implementations will differ. For example in buffered IO, we can setup in such a way that the block plug timeout function runs on the IO completion cpu.

It would be nice to be doing that anyway. But unplug via request submission rather than timeout is fairly common in buffered loads too.

Quoted 7 lines · Nick Piggin

Quoted 3 lines · Suresh B

task by taking away some of its cpu time. With CFS micro accounting, perhaps we can track irq, softirq time and avoid penalizing the running task's cpu time.

But you "penalize" the running task in the completion handler as well anyway.

Yes.

Ingo, in general with CFS micro accounting, we should be able to avoid penalizing the running task by tracking irq/softirq time. Isn't it?

Quoted 2 lines

Doing this with a SCHED_FIFO task is sort of like doing interrupt threading which AFAIK has not been accepted (yet).

I am not recommending SCHED_FIFO. I will take a look at softirq infrastructure for this.

Quoted 5 lines

Quoted 2 lines

This workload is using direct IO and there is no batching at the block layer for direct IO. IO is submitted to the HW as it arrives.

So you aren't putting concurrent requests into the queue? Sounds like userspace should be improved.

Nick remember that there are hundreds of disks in this setup and at an instance, there will be max 1 or 2 requests per disk.

Quoted 6 lines

Quoted 3 lines

It is applicable for both direct IO and buffered IO. But the implementations will differ. For example in buffered IO, we can setup in such a way that the block plug timeout function runs on the IO completion cpu.

It would be nice to be doing that anyway. But unplug via request submission rather than timeout is fairly common in buffered loads too.

Ok. Currently the patch handles both direct and buffered IO. While making improvements to this patch I will make sure that both the paths take advantage of this.

Second experiment which we did was migrating the IO submission to the IO completion cpu. Instead of submitting the IO on the same cpu where the request arrived, in this experiment the IO submission gets migrated to the cpu that is processing IO completions(interrupt). This will minimize the access to remote cachelines (that happens in timers, slab, scsi layers). The IO submission request is forwarded to the kblockd thread on the cpu receiving the interrupts. As part of this, we also made kblockd thread on each cpu as the highest priority thread, so that IO gets submitted as soon as possible on the interrupt cpu with out any delay. On x86_64 SMP platform with 16 cores, this resulted in 2% performance improvement and 3.3% improvement on two node ia64 platform.

Quick and dirty prototype patch(not meant for inclusion) for this io migration experiment is appended to this e-mail.

Observation #1 mentioned above is also applicable to this experiment. CPU's processing interrupts will now have to cater IO submission/processing load aswell.

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

arch/x86/kernel/smp_64.c +2 -0

+asmlinkage void smp_call_function_fast_interrupt(void)
+{

Hi Nick,

Quoted 2 lines · Nick Piggin

arch/x86/kernel/smp_64.c +2 -0

+asmlinkage void smp_call_function_fast_interrupt(void)
+{

[snip]

arch/x86/kernel/smp_64.c void unlock_ipi_call_lock(void) · line 408

Quoted 10 lines

arch/x86/kernel/smp_64.c +10 -0

+       while (!list_empty(&list)) {
+               struct call_single_data *data;
+
+               data = list_entry(list.next, struct call_single_data, list);
+               list_del(&data->list);
+
+               data->func(data->info);
+               if (data->wait) {
+                       smp_mb();
+                       data->wait = 0;

Why do we need smp_mb() here (maybe add a comment to keep Andrew/checkpatch happy)?

Quoted 20 lines · Suresh B

Second experiment which we did was migrating the IO submission to the IO completion cpu. Instead of submitting the IO on the same cpu where the request arrived, in this experiment the IO submission gets migrated to the cpu that is processing IO completions(interrupt). This will minimize the access to remote cachelines (that happens in timers, slab, scsi layers). The IO submission request is forwarded to the kblockd thread on the cpu receiving the interrupts. As part of this, we also made kblockd thread on each cpu as the highest priority thread, so that IO gets submitted as soon as possible on the interrupt cpu with out any delay. On x86_64 SMP platform with 16 cores, this resulted in 2% performance improvement and 3.3% improvement on two node ia64 platform.

Quick and dirty prototype patch(not meant for inclusion) for this io migration experiment is appended to this e-mail.

Observation #1 mentioned above is also applicable to this experiment. CPU's processing interrupts will now have to cater IO submission/processing load aswell.

Quoted 7 lines

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

Hi guys,

Just had another way we might do this. Migrate the completions out to the submitting CPUs rather than migrate submission into the completing CPU.

Hi Nick,

When Matthew was describing this work at an LCA presentation (not sure whether you were at that presentation or not), Zach came up with the idea that allowing the submitting application control the CPU that the io completion processing was occurring would be a good approach to try. That is, we submit a "completion cookie" with the bio that indicates where we want completion to run, rather than dictating that completion runs on the submission CPU.

The reasoning is that only the higher level context really knows what is optimal, and that changes from application to application.

Quoted 10 lines · David Chinner

Hi Nick,

When Matthew was describing this work at an LCA presentation (not sure whether you were at that presentation or not), Zach came up with the idea that allowing the submitting application control the CPU that the io completion processing was occurring would be a good approach to try. That is, we submit a "completion cookie" with the bio that indicates where we want completion to run, rather than dictating that completion runs on the submission CPU.

Quoted 2 lines

The reasoning is that only the higher level context really knows what is optimal, and that changes from application to application.

well.. kinda. One of the really hard parts of the submit/completion stuff is that
the slab/slob/slub/slib allocator ends up basically "cycling" memory through the system;
there's a sink of free memory on all the submission cpus and a source of free memory
on the completion cpu. I don't think applications are capable of working out what is
best in this scenario..

Quoted 14 lines · Arjan van de Ven

Quoted 10 lines · David Chinner

Hi Nick,

When Matthew was describing this work at an LCA presentation (not sure whether you were at that presentation or not), Zach came up with the idea that allowing the submitting application control the CPU that the io completion processing was occurring would be a good approach to try. That is, we submit a "completion cookie" with the bio that indicates where we want completion to run, rather than dictating that completion runs on the submission CPU.

Quoted 2 lines

The reasoning is that only the higher level context really knows what is optimal, and that changes from application to application.

Quoted 9 lines

well.. kinda. One of the really hard parts of the submit/completion stuff is that
the slab/slob/slub/slib allocator ends up basically "cycling" memory through the system;
there's a sink of free memory on all the submission cpus and a source of free memory
on the completion cpu. I don't think applications are capable of working out what is
best in this scenario..

Applications as in "anything that calls submit_bio()". i.e, direct I/O, filesystems, etc. i.e. not userspace but in-kernel applications.

In XFS, simultaneous io completion on multiple CPUs can contribute greatly to contention of global structures in XFS. By controlling where completions are delivered, we can greatly reduce this contention, especially on large, mulitpathed devices that deliver interrupts to multiple CPUs that may be far distant from each other. We have all the state and intelligence necessary to control this sort policy decision effectively.....

You get better behaviour in the slab and page allocators and locality and cache hotness of memory. For example, I guess in a filesystem / pagecache heavy workload, you have to touch each struct page, buffer head, fs private state, and also often have to wake the thread for completion. Much of this data has just been touched at submit time, so doin this on the same CPU is nice...

I'm surprised that the xfs global state bouncing would outweigh the bouncing of all the per-page/block/bio/request/etc data that gets touched during completion. We'll see.

Quoted 6 lines · Nick Piggin

You get better behaviour in the slab and page allocators and locality and cache hotness of memory. For example, I guess in a filesystem / pagecache heavy workload, you have to touch each struct page, buffer head, fs private state, and also often have to wake the thread for completion. Much of this data has just been touched at submit time, so doin this on the same CPU is nice...

[....]

Quoted 3 lines

I'm surprised that the xfs global state bouncing would outweigh the bouncing of all the per-page/block/bio/request/etc data that gets touched during completion. We'll see.

per-page/block.bio/request/etc is local to a single I/O. the only penalty is a cacheline bounce for each of the structures from one CPU to another. That is, there is no global state modified by these completions.

The real issue is metadata. The transaction log I/O completion funnels through a state machine protected by a single lock, which means completions on different CPUs pulls that lock to all completion CPUs. Given that the same lock is used during transaction completion for other state transitions (in task context, not intr), the more cpus active at once touches, the worse the problem gets.

Hi Nick,

When Matthew was describing this work at an LCA presentation (not sure whether you were at that presentation or not), Zach came up with the idea that allowing the submitting application control the CPU that the io completion processing was occurring would be a good approach to try. That is, we submit a "completion cookie" with the bio that indicates where we want completion to run, rather than dictating that completion runs on the submission CPU.

The reasoning is that only the higher level context really knows what is optimal, and that changes from application to application. The "complete on the submission CPU" policy _may_ be more optimal for database workloads, but it is definitely suboptimal for XFS and transaction I/O completion handling because it simply drags a bunch of global filesystem state around between all the CPUs running completions. In that case, we really only want a single CPU to be handling the completions.....

(Zach - please correct me if I've missed anything)

Looking at your patch - if you turn it around so that the "submission CPU" field can be specified as the "completion cpu" then I think the patch will expose the policy knobs needed to do the above.

[ ugh, still jet lagged. ]

Quoted 19 lines

Hi Nick,

When Matthew was describing this work at an LCA presentation (not sure whether you were at that presentation or not), Zach came up with the idea that allowing the submitting application control the CPU that the io completion processing was occurring would be a good approach to try. That is, we submit a "completion cookie" with the bio that indicates where we want completion to run, rather than dictating that completion runs on the submission CPU.

The reasoning is that only the higher level context really knows what is optimal, and that changes from application to application. The "complete on the submission CPU" policy _may_ be more optimal for database workloads, but it is definitely suboptimal for XFS and transaction I/O completion handling because it simply drags a bunch of global filesystem state around between all the CPUs running completions. In that case, we really only want a single CPU to be handling the completions.....

Quoted 1 line

(Zach - please correct me if I've missed anything)

Yeah, I think Nick's patch (and Jens' approach, presumably) is just the sort of thing we were hoping for when discussing this during Matthew's talk.

I was imagining the patch a little bit differently (per-cpu tasks, do a wake_up from the driver instead of cpu nr testing up in blk, work queues, whatever), but we know how to iron out these kinds of details ;).

Quoted 4 lines

Looking at your patch - if you turn it around so that the "submission CPU" field can be specified as the "completion cpu" then I think the patch will expose the policy knobs needed to do the above.

Yeah, that seems pretty straight forward.

We might need some logic for noticing that the desired cpu has been hot-plugged away while the IO was in flight, it occurs to me.

Quoted 3 lines

I was imagining the patch a little bit differently (per-cpu tasks, do a wake_up from the driver instead of cpu nr testing up in blk, work queues, whatever), but we know how to iron out these kinds of details ;).

per-cpu tasks/wq's might be better, it's a little awkward to jump through hoops

Quoted 4 lines · Jens Axboe

Quoted 3 lines

I was imagining the patch a little bit differently (per-cpu tasks, do a wake_up from the driver instead of cpu nr testing up in blk, work queues, whatever), but we know how to iron out these kinds of details ;).

Quoted 3 lines

per-cpu tasks/wq's might be better, it's a little awkward to jump through hoops

one caveat btw; when the multiqueue storage hw becomes available for Linux, we need to figure out how to deal with the preference thing; since there honoring a "non-logical" preference would be quite expensive (it means

you can't make the local submit queues lockless etc etc), so before we go down the road of having widespread APIs for this stuff.. we need to make sure we're not going to do something that's going to be really stupid 6 to 18 months down the road.

Quoted 20 lines · Suresh B

Second experiment which we did was migrating the IO submission to the IO completion cpu. Instead of submitting the IO on the same cpu where the request arrived, in this experiment the IO submission gets migrated to the cpu that is processing IO completions(interrupt). This will minimize the access to remote cachelines (that happens in timers, slab, scsi layers). The IO submission request is forwarded to the kblockd thread on the cpu receiving the interrupts. As part of this, we also made kblockd thread on each cpu as the highest priority thread, so that IO gets submitted as soon as possible on the interrupt cpu with out any delay. On x86_64 SMP platform with 16 cores, this resulted in 2% performance improvement and 3.3% improvement on two node ia64 platform.

Quick and dirty prototype patch(not meant for inclusion) for this io migration experiment is appended to this e-mail.

Observation #1 mentioned above is also applicable to this experiment. CPU's processing interrupts will now have to cater IO submission/processing load aswell.

Quoted 7 lines

Observation #2: This introduces some migration overhead during IO submission. With the current prototype, every incoming IO request results in an IPI and context switch(to kblockd thread) on the interrupt processing cpu. This issue needs to be addressed and main challenge to address is the efficient mechanism of doing this IO migration(how much batching to do and when to send the migrate request?), so that we don't delay the IO much and at the same point, don't cause much overhead during migration.

Hi guys,

Just had another way we might do this. Migrate the completions out to the submitting CPUs rather than migrate submission into the completing CPU.

I've got a basic patch that passes some stress testing. It seems fairly simple to do at the block layer, and the bulk of the patch involves introducing a scalable smp_call_function for it.

Now it could be optimised more by looking at batching up IPIs or optimising the call function path or even mirating the completion event at a different level...

However, this is a first cut. It actually seems like it might be taking slightly more CPU to process block IO (~0.2%)... however, this is on my dual core system that shares an llc, which means that there are very few cache benefits to the migration, but non-zero overhead. So on multisocket systems hopefully it might get to positive territory.

Quoted 16 lines · Nick Piggin

On Fri, Jul 27, 2007 at 06:21:28PM -0700, Suresh B wrote:

Hi guys,

Just had another way we might do this. Migrate the completions out to the submitting CPUs rather than migrate submission into the completing CPU.

I've got a basic patch that passes some stress testing. It seems fairly simple to do at the block layer, and the bulk of the patch involves introducing a scalable smp_call_function for it.

Now it could be optimised more by looking at batching up IPIs or optimising the call function path or even mirating the completion event at a different level...

Quoted 5 lines

However, this is a first cut. It actually seems like it might be taking slightly more CPU to process block IO (~0.2%)... however, this is on my dual core system that shares an llc, which means that there are very few cache benefits to the migration, but non-zero overhead. So on multisocket systems hopefully it might get to positive territory.

That's pretty funny, I did pretty much the exact same thing last week!

The primary difference between yours and mine is that I used a more private interface to signal a softirq raise on another CPU, instead of allocating call data and exposing a generic interface. That put the locking in blk-core instead, turning blk_cpu_done into a structure with a lock and list_head instead of just being a list head, and intercepted at blk_complete_request() time instead of waiting for an already raised softirq on that CPU.

Quoted 17 lines · Jens Axboe

Quoted 16 lines · Nick Piggin

On Fri, Jul 27, 2007 at 06:21:28PM -0700, Suresh B wrote:

Hi guys,

Just had another way we might do this. Migrate the completions out to the submitting CPUs rather than migrate submission into the completing CPU.

I've got a basic patch that passes some stress testing. It seems fairly simple to do at the block layer, and the bulk of the patch involves introducing a scalable smp_call_function for it.

Now it could be optimised more by looking at batching up IPIs or optimising the call function path or even mirating the completion event at a different level...

Quoted 7 lines

Quoted 5 lines

However, this is a first cut. It actually seems like it might be taking slightly more CPU to process block IO (~0.2%)... however, this is on my dual core system that shares an llc, which means that there are very few cache benefits to the migration, but non-zero overhead. So on multisocket systems hopefully it might get to positive territory.

That's pretty funny, I did pretty much the exact same thing last week!

Oh nice ;)

Quoted 7 lines

The primary difference between yours and mine is that I used a more private interface to signal a softirq raise on another CPU, instead of allocating call data and exposing a generic interface. That put the locking in blk-core instead, turning blk_cpu_done into a structure with a lock and list_head instead of just being a list head, and intercepted at blk_complete_request() time instead of waiting for an already raised softirq on that CPU.

Yeah I was looking at that... didn't really want to add the spinlock overhead to the non-migration case. Anyway, I guess that sort of fine implementation details is going to have to be sorted out with results.

As Andi mentions, we can look into making that lockless. For the initial implementation I didn't really care, just wanted something to play with that would nicely allow me to control both the submit and complete side of the affinity issue.

Hi guys,

Just had another way we might do this. Migrate the completions out to the submitting CPUs rather than migrate submission into the completing CPU.