NBD: make nbd default to deadline I/O scheduler
Subsystems (2)
PATCH 1/1 NBD: make nbd default to deadline I/O scheduler
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
--
Paul
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
drivers/block/nbd.c renamed from drivers/block/nbd.c.max_nbd_killed +1 -0
@@ -667,6 +667,7 @@ static int __init nbd_init(void)put_disk (disk); goto out; } +elevator_init (disk->queue,"deadline" ); }if (register_blkdev (NBD_MAJOR ,"nbd" )) {
Quoted 2 lines · On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
so what happens with this patch on cfq-only or as-only kernels?
---
~Randy
Quoted 4 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
Quoted 2 lines · On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
Quoted 1 line
so what happens with this patch on cfq-only or as-only kernels?
I assume the elevator_init() call fails and the default elevator continues to be used. Perhaps an informative printk is needed.
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
Quoted 1 line · On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
The two problems I have experienced (which may already be fixed):
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
There's a similar debian bug that has been filed as well:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=447638
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
--
Paul
Quoted 22 lines · Paul Clements
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
The two problems I have experienced (which may already be fixed):
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
There's a similar debian bug that has been filed as well:
Quoted 7 lines
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
I'm fine with that, it's one of those things we'll do automatically when we have some sort of disk profile system setup. Devices without seek penalties should not use AS or CFQ.
Asking for a non-existing elevator is not an issue, but it may trigger both printks and a switch to another elevator. So if you ask for "deadline" and it's modular, you'll get cfq again if it's the default.
Your patch looks bad though, you forget to exit the old elevator. And you don't check the return value of elevator_init().
All in all, your patch definitely needs more work before it can be included.
Signature · 1 line
Jens Axboe
Quoted 39 lines · Jens Axboe
Quoted 22 lines · Paul Clements
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
The two problems I have experienced (which may already be fixed):
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
There's a similar debian bug that has been filed as well:
Quoted 7 lines
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
I'm fine with that, it's one of those things we'll do automatically when we have some sort of disk profile system setup. Devices without seek penalties should not use AS or CFQ.
Asking for a non-existing elevator is not an issue, but it may trigger both printks and a switch to another elevator. So if you ask for "deadline" and it's modular, you'll get cfq again if it's the default.
Quoted 5 lines
Your patch looks bad though, you forget to exit the old elevator. And you don't check the return value of
elevator_init().All in all, your patch definitely needs more work before it can be included.
argr and please don't CC closed lists on lkml, thanks! it's hugely annoying to everyone that replies in the thread.
Signature · 1 line
Jens Axboe
Quoted 31 lines · Jens Axboe
Quoted 30 lines · Jens Axboe
Quoted 14 lines · Paul Clements
Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
Quoted 1 line · On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
The two problems I have experienced (which may already be fixed):
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
Quoted 7 lines
There's a similar debian bug that has been filed as well:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=447638
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
I'm fine with that, it's one of those things we'll do automatically when we have some sort of disk profile system setup. Devices without seek penalties should not use AS or CFQ.
Asking for a non-existing elevator is not an issue, but it may trigger both printks and a switch to another elevator. So if you ask for "deadline" and it's modular, you'll get cfq again if it's the default.
Quoted 5 lines
Quoted 5 lines
Your patch looks bad though, you forget to exit the old elevator. And you don't check the return value of
elevator_init().All in all, your patch definitely needs more work before it can be included.
Thanks Jens. This one should be better.
--
Paul
NBD doesn't work well with CFQ (or AS) schedulers, so let's default to something else.
The two problems I have experienced with nbd and cfq are:
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
There's a similar debian bug that has been filed as well:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=447638
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
drivers/block/nbd.c renamed from drivers/block/nbd.c.max_nbd_killed +6 -0
@@ -667,6 +667,12 @@ static int __init nbd_init(void)put_disk (disk); goto out; } +if (elevator_init (disk->queue,"deadline" ) !=0 ) { +if (elevator_init (disk->queue,"noop" ) !=0 ) { +put_disk (disk); + goto out; + } + } }if (register_blkdev (NBD_MAJOR ,"nbd" )) {
drivers/block/nbd.c static int __init nbd_init(void) · line 676
Quoted 13 lines · On Fri, 08 Feb 2008 16:23:01 -0500 Paul Clements <[email protected]>…
drivers/block/nbd.c renamed from drivers/block/nbd.c.max_nbd_killed +6
-0@@ -667,6 +667,12 @@ static int __init nbd_init(void)put_disk (disk); goto out; } +if (elevator_init (disk->queue,"deadline" ) !=0 ) { +if (elevator_init (disk->queue,"noop" ) !=0 ) { +put_disk (disk); + goto out; + } + } }
- if the user doesn't have deadline or noop configured, NBD will now fail. That's a non-backward-compatible change.
- when it fails, it will fail silently. Puzzled and angry users.
- when it fails, it will inappropriately return -ENOMEM.
Quoted 8 lines · Paul Clements
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Fri, 8 Feb 2008 09:33:41 -0800 Randy Dunlap <[email protected]> wrote:
Quoted 2 lines · On Fri, 08 Feb 2008 11:47:42 -0500 Paul Clements wrote:
There have been numerous reports of problems with nbd and cfq. Deadline gives better performance for nbd, anyway, so let's use it by default.
Quoted 8 lines
Quoted 4 lines
Please define "problems". If it's just "slowness" then we can live with that, but I'd hope that Jens is aware and that it's understood.
It it's "hangs" or "oopses" then we panic.
The two problems I have experienced (which may already be fixed):
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
It has been fixed in RHEL5 (but not yet released AFAIK):
https://bugzilla.redhat.com/show_bug.cgi?id=241540#c43
On a slightly related performance note; I'm seeing that for NBD devices both max_hw_sectors_kb and max_sectors_kb are 127. If I set it to be 256 with the following patch I see a 25% improvement in overall throughput:
drivers/block/nbd.c +1 -0
@@ -764,6 +764,7 @@ static int __init nbd_init(void)put_disk (disk); goto out; } +blk_queue_max_sectors (disk->queue,512 ); }if (register_blkdev (NBD_MAJOR ,"nbd" )) {
Any chance we can take steps to make NBD not be artificially slow? Any other recommendations for improving NBD throughput?
regards,
Mike
Take 3...this should address all the issues.
NBD doesn't work well with CFQ (or AS) schedulers, so let's default to something else.
The two problems I have experienced with nbd and cfq are:
1) nbd hangs with cfq on RHEL 5 (2.6.18) -- this may well have been fixed
There's a similar debian bug that has been filed as well:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=447638
There have been posts to nbd-general mailing list about problems with cfq and nbd also.
2) nbd performs about 10% better (the last time I tested) with deadline vs. cfq (the overhead of cfq doesn't provide much advantage to nbd [not being a real disk], and you end up going through the I/O scheduler on the nbd server anyway, so it makes sense that deadline is better with nbd)
drivers/block/nbd.c renamed from drivers/block/nbd.c.max_nbd_killed +6 -0
@@ -654,6 +654,7 @@ static int __init nbd_init(void)for (i =0 ; i < nbds_max; i++) {struct gendisk *disk =alloc_disk (1 ); +elevator_t *old_e;if (!disk) goto out; nbd_dev[i].disk = disk;
@@ -667,6 +668,11 @@ static int __init nbd_init(void)put_disk (disk); goto out; } + old_e = disk->queue->elevator; +if (elevator_init (disk->queue,"deadline" ) ==0 || +elevator_init (disk->queue,"noop" ) ==0 ) { +elevator_exit (old_e); + } }if (register_blkdev (NBD_MAJOR ,"nbd" )) {
drivers/block/nbd.c static int __init nbd_init(void) · line 676
Quoted 6 lines · On Sat, 09 Feb 2008 08:30:40 -0500 Paul Clements <[email protected]>…
drivers/block/nbd.c renamed from drivers/block/nbd.c.max_nbd_killed +5
-0+ old_e = disk->queue->elevator; +if (elevator_init (disk->queue,"deadline" ) ==0 || +elevator_init (disk->queue,"noop" ) ==0 ) { +elevator_exit (old_e); + } }
afacit elevator_init() will not trigger a request_module(). And you really do want to trigger the request_module() here. Perhaps the block layer should provide a means of doing so?
Quoted 13 lines · Andrew Morton
Review
Quoted 6 lines · On Sat, 09 Feb 2008 08:30:40 -0500 Paul Clements <[email protected]>…
(quoted diff) +5
-0+ old_e = disk->queue->elevator; + if (elevator_init(disk->queue, "deadline") == 0 || + elevator_init(disk->queue, "noop") == 0) { + elevator_exit(old_e); + } }afacit
elevator_init()will not trigger arequest_module(). And you really do want to trigger therequest_module()here. Perhaps the block layer should provide a means of doing so?
Good point, I think elevator_get() should do that automatically. Does this look sane?
block/elevator.c +15 -0
@@ -134,6 +134,21 @@ static struct elevator_type *elevator_get(const char *name)spin_lock (&elv_list_lock); e =elevator_find (name); +if (!e) { +char elv[ELV_NAME_MAX +strlen ("-iosched" )]; + +spin_unlock (&elv_list_lock); + +if (!strcmp (name,"anticipatory" )) +sprintf (elv,"as-iosched" ); +else +sprintf (elv,"%s-iosched" , name); + +request_module (elv); +spin_lock (&elv_list_lock); + e =elevator_find (name); + } +if (e && !try_module_get (e->elevator_owner)) e =NULL ;
Signature · 1 line
Jens Axboe
Quoted 33 lines · On Mon, 18 Feb 2008 19:16:30 +0100 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Review
Quoted 6 lines · On Sat, 09 Feb 2008 08:30:40 -0500 Paul Clements <[email protected]>…
(quoted diff) +5
-0+ old_e = disk->queue->elevator; + if (elevator_init(disk->queue, "deadline") == 0 || + elevator_init(disk->queue, "noop") == 0) { + elevator_exit(old_e); + } }afacit
elevator_init()will not trigger arequest_module(). And you really do want to trigger therequest_module()here. Perhaps the block layer should provide a means of doing so?Good point, I think
elevator_get()should do that automatically. Does this look sane?block/elevator.c +7
-0@@ -134,6 +134,21 @@ static struct elevator_type *elevator_get(const char *name)spin_lock (&elv_list_lock); e =elevator_find (name); +if (!e) { +char elv[ELV_NAME_MAX +strlen ("-iosched" )]; + +spin_unlock (&elv_list_lock); + +if (!strcmp (name,"anticipatory" )) +sprintf (elv,"as-iosched" );
block/elevator.c static struct elevator_type *elevator_get(const char *name) · line 153
Quoted 10 lines
block/elevator.c +8
-0+else +sprintf (elv,"%s-iosched" , name); + +request_module (elv); +spin_lock (&elv_list_lock); + e =elevator_find (name); + } +if (e && !try_module_get (e->elevator_owner)) e =NULL ;
Looks nice and simple. There might be some of the usual ordering problems when this is called during boot, maybe is-initramfs-available-yet problems, etc. But it's unlikely to make things regress from where they are now.
Should we emit a warning if the desired elevator wasn't available?
Quoted 41 lines · Andrew Morton
Quoted 39 lines · On Mon, 18 Feb 2008 19:16:30 +0100 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Review
Quoted 6 lines · On Sat, 09 Feb 2008 08:30:40 -0500 Paul Clements <[email protected]>…
(quoted diff) +5
-0+ old_e = disk->queue->elevator; + if (elevator_init(disk->queue, "deadline") == 0 || + elevator_init(disk->queue, "noop") == 0) { + elevator_exit(old_e); + } }afacit
elevator_init()will not trigger arequest_module(). And you really do want to trigger therequest_module()here. Perhaps the block layer should provide a means of doing so?Good point, I think
elevator_get()should do that automatically. Does this look sane?block/elevator.c +13
-0@@ -134,6 +134,21 @@ static struct elevator_type *elevator_get(const char *name)spin_lock (&elv_list_lock); e =elevator_find (name); +if (!e) { +char elv[ELV_NAME_MAX +strlen ("-iosched" )]; + +spin_unlock (&elv_list_lock); + +if (!strcmp (name,"anticipatory" )) +sprintf (elv,"as-iosched" ); +else +sprintf (elv,"%s-iosched" , name); + +request_module (elv); +spin_lock (&elv_list_lock); + e =elevator_find (name);
Quoted 8 lines
block/elevator.c static struct elevator_type *elevator_get(const char *name) · line 153
Quoted 4 lines
block/elevator.c +2
-0+ } +if (e && !try_module_get (e->elevator_owner)) e =NULL ;Looks nice and simple. There might be some of the usual ordering problems when this is called during boot, maybe is-initramfs-available-yet problems, etc. But it's unlikely to make things regress from where they are now.
Isn't request_module() and below robust enough to handle that?
Quoted 1 line
Should we emit a warning if the desired elevator wasn't available?
Hmm, not sure. Either the request came from a driver, in which case it'll be notified that we could not load that elevator. Or it'll come through sysfs online switching, in which case we already print a warning.
Signature · 1 line
Jens Axboe
Quoted 51 lines · Jens Axboe
Quoted 41 lines · Andrew Morton
Quoted 39 lines · On Mon, 18 Feb 2008 19:16:30 +0100 Jens Axboe <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Review
Quoted 6 lines · On Sat, 09 Feb 2008 08:30:40 -0500 Paul Clements <[email protected]>…
(quoted diff) +5
-0+ old_e = disk->queue->elevator; + if (elevator_init(disk->queue, "deadline") == 0 || + elevator_init(disk->queue, "noop") == 0) { + elevator_exit(old_e); + } }afacit
elevator_init()will not trigger arequest_module(). And you really do want to trigger therequest_module()here. Perhaps the block layer should provide a means of doing so?Good point, I think
elevator_get()should do that automatically. Does this look sane?block/elevator.c +13
-0@@ -134,6 +134,21 @@ static struct elevator_type *elevator_get(const char *name)spin_lock (&elv_list_lock); e =elevator_find (name); +if (!e) { +char elv[ELV_NAME_MAX +strlen ("-iosched" )]; + +spin_unlock (&elv_list_lock); + +if (!strcmp (name,"anticipatory" )) +sprintf (elv,"as-iosched" ); +else +sprintf (elv,"%s-iosched" , name); + +request_module (elv); +spin_lock (&elv_list_lock); + e =elevator_find (name);Quoted 8 lines
block/elevator.c static struct elevator_type *elevator_get(const char *name) · line 153
Quoted 4 lines
block/elevator.c +2
-0+ } +if (e && !try_module_get (e->elevator_owner)) e =NULL ;Looks nice and simple. There might be some of the usual ordering problems when this is called during boot, maybe is-initramfs-available-yet problems, etc. But it's unlikely to make things regress from where they are now.
Quoted 1 line
Isn't
request_module()and below robust enough to handle that?
BTW, I've verified that it works as expected (at least after boot):
carl:/sys/block/sda/queue # cat scheduler
noop [cfq]
carl:/sys/block/sda/queue # echo anticipatory > scheduler carl:/sys/block/sda/queue # dmesg
[...]
io scheduler anticipatory registered carl:/sys/block/sda/queue # cat scheduler
noop cfq [anticipatory]
So it properly loads as-iosched instead of failing, like it would have done before and required the user to do a modprobe as-iosched first.
carl:/sys/block/sda/queue # echo foobar > scheduler -bash: echo: write error: Invalid argument carl:/sys/block/sda/queue # dmesg
[...]
elevator: type foobar not found
Signature · 1 line
Jens Axboe
Quoted 23 lines · On Tue, 19 Feb 2008 10:24:28 +0100 Jens Axboe <[email protected]> wrote:
Quoted 10 lines · Jens Axboe
Quoted 7 lines · Andrew Morton
block/elevator.c static struct elevator_type *elevator_get(const char *name) · line 153
Quoted 3 lines
block/elevator.c +1
-0+if (e && !try_module_get (e->elevator_owner)) e =NULL ;Looks nice and simple. There might be some of the usual ordering problems when this is called during boot, maybe is-initramfs-available-yet problems, etc. But it's unlikely to make things regress from where they are now.
Isn't
request_module()and below robust enough to handle that?BTW, I've verified that it works as expected (at least after boot):
carl:/sys/block/sda/queue # cat scheduler
noop [cfq]
carl:/sys/block/sda/queue # echo anticipatory > scheduler carl:/sys/block/sda/queue # dmesg
[...]
io scheduler anticipatory registered carl:/sys/block/sda/queue # cat scheduler
noop cfq [anticipatory]
Quoted 8 lines
So it properly loads as-iosched instead of failing, like it would have done before and required the user to do a modprobe as-iosched first.
carl:/sys/block/sda/queue # echo foobar > scheduler -bash: echo: write error: Invalid argument carl:/sys/block/sda/queue # dmesg
[...]
elevator: type foobar not found
Looks promising - let's run with it. If there _are_ startup ordering problems then we won't be any worse off than we are now.
otoh, the system must have _some_ io scheduler installed when talking to disks, so perhaps there won't be any such problems at all.
Quoted 28 lines · Andrew Morton
Quoted 26 lines · On Tue, 19 Feb 2008 10:24:28 +0100 Jens Axboe <[email protected]> wrote:
Quoted 10 lines · Jens Axboe
Quoted 7 lines · Andrew Morton
block/elevator.c static struct elevator_type *elevator_get(const char *name) · line 153
Quoted 3 lines
block/elevator.c +1
-0+if (e && !try_module_get (e->elevator_owner)) e =NULL ;Looks nice and simple. There might be some of the usual ordering problems when this is called during boot, maybe is-initramfs-available-yet problems, etc. But it's unlikely to make things regress from where they are now.
Isn't
request_module()and below robust enough to handle that?BTW, I've verified that it works as expected (at least after boot):
carl:/sys/block/sda/queue # cat scheduler
noop [cfq]
carl:/sys/block/sda/queue # echo anticipatory > scheduler carl:/sys/block/sda/queue # dmesg
[...]
io scheduler anticipatory registered carl:/sys/block/sda/queue # cat scheduler
noop cfq [anticipatory]So it properly loads as-iosched instead of failing, like it would have done before and required the user to do a modprobe as-iosched first.
Quoted 8 lines
Quoted 5 lines
carl:/sys/block/sda/queue # echo foobar > scheduler -bash: echo: write error: Invalid argument carl:/sys/block/sda/queue # dmesg
[...]
elevator: type foobar not foundLooks promising - let's run with it. If there _are_ startup ordering problems then we won't be any worse off than we are now.
I've merged it for inclusion, since it tests fine here.
Quoted 2 lines
otoh, the system must have _some_ io scheduler installed when talking to disks, so perhaps there won't be any such problems at all.
We are guarenteed to always have noop available, since you cannot de-select that.
Signature · 1 line
Jens Axboe