Skip to contentSkip to conversation

Re: [Fwd: Block driver freezes when using CFQ]

Fwd: Block driver freezes when using CFQ Jens Axboe 2 participants 3 messages Oct 30, 2006 – Oct 31, 2006 1 message not in the archive

lore
Share & tools
lore

Jens Axboe

author
#1 lore raw
In reply to a message not in the archive
Quoted 27 lines · Ravi Krishnamurthy

Hi all,

I have written a block driver that registers a virtual device and
routes requests to appropriate real devices after some re-mapping of the requests. I am testing the driver by creating a filesystem on the virtual device and copying a large number of files on to it. The test causes the device to become unresponsive after some time. After some debugging, I noticed that this happens only if the I/O scheduler being used is CFQ. I have not had any trouble if the scheduler is noop, anticipatory or deadline. The problem occurs on all the kernels I have tested - 2.6.18-rc2, 2.6.18-rc4, 2.6.19-rc3.

Below are some details about the driver and what I have observed during testing:

The request function registered by my driver is a simple loop -

  while ((req = elv_next_request(q))) {
        blkdev_dequeue_request(req);

/*
Add request to an internal queue for further processing Wake up thread to start processing the queue Update some variables for book-keeping
*/
}

Quoted 8 lines

Completed requests are handled in a different thread -
while (work to be done) {
/*
Dequeue completed requests from internal queue Call end_that_request_first() and end_that_request_last() Update some variables for book-keeping
*/
}

The io scheduler is not obligated to recall your request handling function, _unless_ you have no pending io at the point where elv_next_request() returns NULL but there are things pending. IOW, when you complete your requests you want to just recall your request handling function. Just insert something ala:

        if (elv_next_request(q))
                q->request_fn(q);

when you are done completing requests.

Does that fix it?

Signature · 1 line
Jens Axboe
#2 lore raw

Re: Block driver freezes when using CFQ

Quoted 21 lines · Jens Axboe
Quoted 12 lines · Ravi Krishnamurthy

Hi all,

I have written a block driver that registers a virtual device and
routes requests to appropriate real devices after some re-mapping of the requests. I am testing the driver by creating a filesystem on the virtual device and copying a large number of files on to it. The test causes the device to become unresponsive after some time. After some debugging, I noticed that this happens only if the I/O scheduler being used is CFQ. I have not had any trouble if the scheduler is noop, anticipatory or deadline. The problem occurs on all the kernels I have tested - 2.6.18-rc2, 2.6.18-rc4, 2.6.19-rc3.

The io scheduler is not obligated to recall your request handling
function, _unless_ you have no pending io at the point where
elv_next_request() returns NULL but there are things pending. IOW, when you complete your requests you want to just recall your request handling function. Just insert something ala:

Quoted 6 lines
        if (elv_next_request(q))
                q->request_fn(q);

when you are done completing requests.

Does that fix it?

I haven't had a chance to test this fix. A workaround I had tried was to insert these lines at the end of the request function:

       if (! elv_queue_empty(q))
            blk_plug_device(q);

This worked for me. So I assume the fix you have suggested will surely work.

I am curious to know why the problem does not occur when I am using the anticipatory scheduler. Also, in the suggested fix, is it guaranteed that elv_next_request() will not return NULL as long as the elevator queue is not empty?

Thanks,
Ravi.

author
#3 lore raw

Re: Block driver freezes when using CFQ

Quoted 31 lines · Ravi Krishnamurthy
Quoted 23 lines · Jens Axboe
Quoted 12 lines · Ravi Krishnamurthy

Hi all,

I have written a block driver that registers a virtual device and
routes requests to appropriate real devices after some re-mapping of the requests. I am testing the driver by creating a filesystem on the virtual device and copying a large number of files on to it. The test causes the device to become unresponsive after some time. After some debugging, I noticed that this happens only if the I/O scheduler being used is CFQ. I have not had any trouble if the scheduler is noop, anticipatory or deadline. The problem occurs on all the kernels I have tested - 2.6.18-rc2, 2.6.18-rc4, 2.6.19-rc3.

The io scheduler is not obligated to recall your request handling function, _unless_ you have no pending io at the point where elv_next_request() returns NULL but there are things pending. IOW, when you complete your requests you want to just recall your request handling
function. Just insert something ala:

Quoted 6 lines
       if (elv_next_request(q))
               q->request_fn(q);

when you are done completing requests.

Does that fix it?

Quoted 7 lines

I haven't had a chance to test this fix. A workaround I had tried was to insert these lines at the end of the request function:

       if (! elv_queue_empty(q))
            blk_plug_device(q);

This worked for me. So I assume the fix you have suggested will surely work.

You don't want to do that. It is the duty of the plugger to unplug the device again, and in your case that is probably deferred to the timer auto-unplug. So don't involve plugging, it's a seperate thing. Just leave the request function when elv_next_request(), and always recall it when you are done completing requests.

Quoted 4 lines

I am curious to know why the problem does not occur when I am using the anticipatory scheduler. Also, in the suggested fix, is it guaranteed that elv_next_request() will not return NULL as long as the elevator queue is not empty?

Perhaps it recalls ->request_fn() more often than it should. If you call

Signature · 1 line
Jens Axboe