Block device strategy and requests
I'm designing a block device driver for a high performance disk subsystem with unusual characteristics. To what extent is the limited number of "struct request"s (128 by default) necessary for back-pressure? With this I/O subsystem it would be possible for the strategy function to rip the requests from the request list straight away, arrange for the I/Os to be done to/from the buffer_heads (with no additional state required) with no memory "leak". This would effectively mean that the only limit on the number of I/Os queued would be the number of buffer_heads allocated; not a fixed number of "struct request"s in flight. Is this reasonable or does any memory or resource balancing depend on the number of I/Os outstanding being bounded?
Also, there is a lot of flexibility in how often interrupts are sent to mark the buffer_heads up-to-date. (With the requests pulled straight off the queue, the job of end_that_request_first() in doing the linked list updates and bh->b_end_io() callbacks would be done by the interrupt routine directly.) At one extreme, I could take an interrupt for each 4K block issued and mark it up-to-date very quickly making for very low-latency I/O but a very large interrupt rate when I/O throughput is high. The alternative would be to arrange for an interrupt every n buffer_heads (or based on some other criterion) and only take an interrupt and mark buffers up-to-date on each of those). Are there any rules of thumb on which is best or doesn't it matter too much?
--Malcolm
Signature · 3 lines
Malcolm Beattie <[email protected]> Unix Systems Programmer Oxford University Computing Services
Quoted 4 lines · Malcolm Beattie
I'm designing a block device driver for a high performance disk subsystem with unusual characteristics. To what extent is the limited number of "struct request"s (128 by default) necessary for back-pressure? With this I/O subsystem it would be possible for the
Not at all
Quoted 8 lines
strategy function to rip the requests from the request list straight away, arrange for the I/Os to be done to/from the
buffer_heads(with no additional state required) with no memory "leak". This would effectively mean that the only limit on the number of I/Os queued would be the number ofbuffer_headsallocated; not a fixed number of "struct request"s in flight. Is this reasonable or does any memory or resource balancing depend on the number of I/Os outstanding being bounded?
The requests need not be bounded, as long as the buffer_heads are. I don't see how the above scheme differs from some of the drivers that are currently in the tree though?
Quoted 4 lines
Also, there is a lot of flexibility in how often interrupts are sent to mark the
buffer_headsup-to-date. (With the requests pulled straight off the queue, the job ofend_that_request_first()in doing the linked list updates andbh->b_end_io()callbacks would be done by
Quoted 8 lines
the interrupt routine directly.) At one extreme, I could take an interrupt for each 4K block issued and mark it up-to-date very quickly making for very low-latency I/O but a very large interrupt rate when I/O throughput is high. The alternative would be to arrange for an interrupt every n
buffer_heads(or based on some other criterion) and only take an interrupt and mark buffers up-to-date on each of those). Are there any rules of thumb on which is best or doesn't it matter too much?
An interrupt per request would give you anywhere between 4kB and XXXkB transfer per interrupt, depending on what you set your max_sectors to. Going bigger than that probably won't make a whole lot of sense, and you would have to do additional foot-work to make it happen. In theory. Only real-life testing can tell you for sure, and how big requests you get depends heavily on the workload so there will be no one true answer for this.
Signature · 1 line
Jens Axboe
Quoted 13 lines · Malcolm Beattie
I'm designing a block device driver for a high performance disk subsystem with unusual characteristics. To what extent is the limited number of "struct request"s (128 by default) necessary for back-pressure? With this I/O subsystem it would be possible for the strategy function to rip the requests from the request list straight away, arrange for the I/Os to be done to/from the
buffer_heads(with no additional state required) with no memory "leak". This would effectively mean that the only limit on the number of I/Os queued would be the number ofbuffer_headsallocated; not a fixed number of "struct request"s in flight. Is this reasonable or does any memory or resource balancing depend on the number of I/Os outstanding being bounded?
Quoted 10 lines
Also, there is a lot of flexibility in how often interrupts are sent to mark the
buffer_headsup-to-date. (With the requests pulled straight off the queue, the job ofend_that_request_first()in doing the linked list updates andbh->b_end_io()callbacks would be done by the interrupt routine directly.) At one extreme, I could take an interrupt for each 4K block issued and mark it up-to-date very quickly making for very low-latency I/O but a very large interrupt rate when I/O throughput is high. The alternative would be to arrange for an interrupt every nbuffer_heads(or based on some other criterion) and only take an interrupt and mark buffers up-to-date on
I believe it is common practice for this type of problem is to mix both approaches:
Wait for a certain number of requests *OR* a timeout, whichever comes first. Then, if there's not much IO, things are still guaranteed to be updated reasonably
quickly. If io is heavy,
then things will be updated in large chunks, becoming more efficient (fewer interrupts) when it is needed most.
Quoted 4 lines
each of those). Are there any rules of thumb on which is best or doesn't it matter too much?