Skip to contentSkip to conversation

Block driver freezes when using CFQ

DISCUSSION Ravi Krishnamurthy 2 participants 2 messages Oct 28, 2006 – Oct 30, 2006

lore
Share & tools
lore
#0 lore raw

Hi all,

I have written a block driver that registers a virtual device and
routes requests to appropriate real devices after some re-mapping of the requests. I am testing the driver by creating a filesystem on the virtual device and copying a large number of files on to it. The test causes the device to become unresponsive after some time. After some debugging, I noticed that this happens only if the I/O scheduler being used is CFQ. I have not had any trouble if the scheduler is noop, anticipatory or deadline. The problem occurs on all the kernels I have tested - 2.6.18-rc2, 2.6.18-rc4, 2.6.19-rc3.

Below are some details about the driver and what I have observed during testing:

The request function registered by my driver is a simple loop -

  while ((req = elv_next_request(q))) {
        blkdev_dequeue_request(req);

/*
Add request to an internal queue for further processing Wake up thread to start processing the queue Update some variables for book-keeping
*/
}

Completed requests are handled in a different thread -
while (work to be done) {
/*
Dequeue completed requests from internal queue Call end_that_request_first() and end_that_request_last() Update some variables for book-keeping
*/
}

Several times during the test run, the while() loop in the request function comes out without dequeuing any request even though the elevator queue is not empty. (Confirmed by printing the return value of elv_queue_empty(), and the values of q->rq.count[] outside the loop). After one such occurrence, the request function is not called at all and the device becomes unresponsive.
I added some code that lets me trigger the request function from userspace. If I nudge the driver this way, I/Os continue for a short while and stop again.

Since CFQ is the default I/O scheduler in current kernels, it has been widely used and tested. So I suspect I am not doing something right in my driver. Since the driver works well with the other schedulers, is there something CFQ-specific that I should take care of?

Please Cc me on the responses since I am not subscribed to lkml.

Thanks,
Ravi.

#1 lore raw

Hello, Ravi.

First of all, it usually attracts more people if you include full source code of something runnable.

Ravi Krishnamurthy wrote:
[--snip--]

Quoted 4 lines

Several times during the test run, the while() loop in the request function comes out without dequeuing any request even though the elevator queue is not empty. (Confirmed by printing the return value of elv_queue_empty(), and the values of q->rq.count[] outside the loop).

Yeap, both cfq and anticipatory pause queue processing to improve performance. This is a bit counter-intuitive at first but the loooong seek time justifies such pauses if they can reduce seeks.

Quoted 10 lines

After one such occurrence, the request function is not called at all and the device becomes unresponsive.
I added some code that lets me trigger the request function from userspace. If I nudge the driver this way, I/Os continue for a short while and stop again.

Since CFQ is the default I/O scheduler in current kernels, it has been widely used and tested. So I suspect I am not doing something right in my driver. Since the driver works well with the other schedulers, is there something CFQ-specific that I should take care of?

After such pauses, cfq does the needed 'nudging' by itself. cfq has changed quite a bit so I might be mistaken but such 'nudging' ends up calling blk_start_queueing() which either runs request_fn directly or unplug the queue if plugged. So, does your driver's queue have proper unplug function? How is your queue initialized? (you can see why it's much better to post full working source.)

Signature · 1 line
tejun