block: allow initialization of previously allocated request_queue
Subsystems (2)
RFC PATCH 1/2 block: allow initialization of previously allocated request_queue
blk_init_queue() allocates the request_queue structure and then initializes it as needed (request_fn, elevator, etc).
Split initialization out to blk_init_allocated_queue_node. Introduce blk_init_allocated_queue wrapper function to model existing blk_init_queue and blk_init_queue_node interfaces.
Export elv_register_queue to allow a newly added elevator to be registered with sysfs. Export elv_unregister_queue for symmetry.
These changes allow DM to initialize a device's request_queue with more precision. In particular, DM no longer unconditionally initializes a full request_queue (elevator et al). It only does so for a request-based DM device.
--- block/blk-core.c | 18 +++++++++++++++++- block/elevator.c | 2 ++ include/linux/blkdev.h | 5 +++++ 3 files changed, 24 insertions(+), 1 deletions(-)
block/blk-core.c +17 -1
@@ -567,7 +567,23 @@ struct request_queue * blk_init_queue_node(request_fn_proc *rfn, spinlock_t *lock, int node_id) {struct request_queue *q =blk_alloc_queue_node (GFP_KERNEL , node_id); + +return blk_init_allocated_queue_node (q, rfn, lock, node_id); +} +EXPORT_SYMBOL (blk_init_queue_node); +struct request_queue * +blk_init_allocated_queue (struct request_queue *q,request_fn_proc *rfn, +spinlock_t *lock) +{ +return blk_init_allocated_queue_node (q, rfn, lock,-1 ); +} +EXPORT_SYMBOL (blk_init_allocated_queue); + +struct request_queue * +blk_init_allocated_queue_node (struct request_queue *q,request_fn_proc *rfn, +spinlock_t *lock,int node_id) +{if (!q)return NULL ;
@@ -601,7 +617,7 @@ blk_init_queue_node(request_fn_proc *rfn, spinlock_t *lock, int node_id)blk_put_queue (q);return NULL ; } -EXPORT_SYMBOL (blk_init_queue_node); +EXPORT_SYMBOL (blk_init_allocated_queue_node);int blk_get_queue (struct request_queue *q) {
block/elevator.c +2 -0
@@ -921,6 +921,7 @@ int elv_register_queue(struct request_queue *q) }return error; } +EXPORT_SYMBOL_GPL (elv_register_queue);static void __elv_unregister_queue (struct elevator_queue *e) {
@@ -933,6 +934,7 @@ void elv_unregister_queue(struct request_queue *q)if (q)__elv_unregister_queue (q->elevator); } +EXPORT_SYMBOL_GPL (elv_unregister_queue);void elv_register (struct elevator_type *e) {
include/linux/blkdev.h +5 -0
@@ -917,7 +917,12 @@ extern void blk_abort_queue(struct request_queue *);*/ extern struct request_queue *blk_init_queue_node (request_fn_proc *rfn,spinlock_t *lock,int node_id); +extern struct request_queue *blk_init_allocated_queue_node (struct request_queue *, +request_fn_proc *, +spinlock_t *,int node_id);extern struct request_queue *blk_init_queue (request_fn_proc *,spinlock_t *); +extern struct request_queue *blk_init_allocated_queue (struct request_queue *, +request_fn_proc *,spinlock_t *);extern void blk_cleanup_queue (struct request_queue *);extern void blk_queue_make_request (struct request_queue *,make_request_fn *);extern void blk_queue_bounce_limit (struct request_queue *,u64 );
Signature · 1 line
1.6.6.1
RFC PATCH 2/2 dm: only initialize full request_queue for request-based device
Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.
dm_init_request_based_queue is now called at the end of a request-based DM device's table load.
Otherwise bio-based DM devices will have an elevator, request_fn, etc. As a result bio-based DM devices had an 'iosched/' tree as well as an 'elevator' other than "none" in sysfs.
--- drivers/md/dm-table.c | 5 ++++ drivers/md/dm.c | 52 ++++++++++++++++++++++++++++++++----------------- drivers/md/dm.h | 2 + 3 files changed, 41 insertions(+), 18 deletions(-)
drivers/md/dm-table.c +5 -0
@@ -829,6 +829,11 @@ int dm_table_set_type(struct dm_table *t)return -EINVAL ; } +if (!dm_init_request_based_queue (t->md)) { +DMWARN ("Cannot initialize queue for Request-based dm" ); +return -EINVAL ; + } + t->type =DM_TYPE_REQUEST_BASED ;return 0 ;
drivers/md/dm.c +34 -18
73 lines not on this page · raw message
drivers/md/dm.h +2 -0
@@ -113,6 +113,8 @@ void dm_sysfs_exit(struct mapped_device *md);struct kobject *dm_kobject (struct mapped_device *md);struct mapped_device *dm_get_from_kobject (struct kobject *kobj); +int dm_init_request_based_queue (struct mapped_device *md); +/* * Targets for linear and striped mappings */
Signature · 1 line
1.6.6.1
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 4 lines · Mike Snitzer
Revert back to only allocating a minimalist
request_queuestructure initially (needed for both bio and request-based DM). Initialization of a fullrequest_queue(request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.
Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)
As I mentioned on the last email of the thread above (see below), specifying device type at the device creation time by userspace tools should make dm code very simple. So that may be a better approach.
Quoted 11 lines
By the way, another approach to optimizing the memory usage would be to determine whether the dm device is bio-based or request-based at the device creation time, instead of the table binding time. We want the delayed allocation, since kernel can't decide the device type until the first table is bound because of the auto-detection mechanism. The auto-detection is good for keeping compatibility with existing user-space tools. But once user-space tools are changed to specify device type at the device creation time, we can eventually remove the auto-detection.
Then, kernel can decide device type inalloc_dev(), so
the initialization code in kernel will become very simple.
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 8 lines · Kiyoshi Ueda
Hi Mike,
Quoted 4 lines · Mike Snitzer
Revert back to only allocating a minimalist
request_queuestructure initially (needed for both bio and request-based DM). Initialization of a fullrequest_queue(request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.
Quoted 5 lines
Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)
Thanks for the reference. I completely forgot about that thread (even though I responded to Nikanth's patches in detail! :)
It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.
I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Interestingly, having to revisit this issue (forgetting that this line of work was already explored) I came up with roughly the same type of change to the block layer as Nikanth's 1/2 patch. The difference being my blk_init_allocated_queue is more minimalist because the block code has evolved to allow this change to be more natural.
Similarly, my proposed DM changes are also quite natural. By using dm_table_set_type() as the hook to initialize the request-based DM device's elevator we perform allocations during table load.
Having just looked at Nikanth's proposed DM patch 2/2 again it shows that blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.
Quoted 18 lines
As I mentioned on the last email of the thread above (see below), specifying device type at the device creation time by userspace tools should make dm code very simple. So that may be a better approach.
Quoted 11 lines
By the way, another approach to optimizing the memory usage would be to determine whether the dm device is bio-based or request-based at the device creation time, instead of the table binding time. We want the delayed allocation, since kernel can't decide the device type until the first table is bound because of the auto-detection mechanism. The auto-detection is good for keeping compatibility with existing user-space tools. But once user-space tools are changed to specify device type at the device creation time, we can eventually remove the auto-detection.
Then, kernel can decide device type inalloc_dev(), so
the initialization code in kernel will become very simple.Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 20 lines · Mike Snitzer
Quoted 8 lines · Kiyoshi Ueda
Hi Mike,
Quoted 4 lines · Mike Snitzer
Revert back to only allocating a minimalist
request_queuestructure initially (needed for both bio and request-based DM). Initialization of a fullrequest_queue(request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.Quoted 5 lines
Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)Thanks for the reference. I completely forgot about that thread (even though I responded to Nikanth's patches in detail! :)
Quoted 7 lines
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.
By the way, your current patch has a problem below. It needs to be fixed at least.
Quoted 3 lines
Similarly, my proposed DM changes are also quite natural. By using
dm_table_set_type()as the hook to initialize the request-based DM device's elevator we perform allocations during table load.
Your patch initializes queue everytime request-based table is loaded. I think that could cause a problem (although I haven't tested your patch yet).
Also, as you know, the table load can be canceled.
So initializing queue at table loading time may cause some weird behaviors for users. For example,
# dmsetup create --notable bio-based
# echo "0 100 multipath ..." | dmsetup load bio-based
# echo "0 100 linear ..." | dmsetup load bio-based
# dmsetup resume bio-based
# ls /sys/block/dm-0/queue/
... iosched ...
If you (and Alasdair) think this behavior is acceptable, it might
be OK. I just feel it's weird though...Quoted 4 lines
Having just looked at Nikanth's proposed DM patch 2/2 again it shows that
blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.
Right, in general.
However, in this special case, I think initializing queue (allocating memory) during resume should be OK as I mentioned like below in:
http://marc.info/?l=linux-kernel&m=124999806420663&w=2
> Generally, dm must not allocate memory during resume because > it may cause a deadlock in no memory situation.
> However, there is no I/O on this device at this point, > so the allocation should be ok for this special case. > I think some comments are needed here to describe that.
So you should be able to take this approach.
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 11 lines · Kiyoshi Ueda
Hi Mike,
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.
OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
Quoted 2 lines
By the way, your current patch has a problem below. It needs to be fixed at least.
Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).
Quoted 7 lines
Quoted 3 lines
Similarly, my proposed DM changes are also quite natural. By using
dm_table_set_type()as the hook to initialize the request-based DM device's elevator we perform allocations during table load.Your patch initializes queue everytime request-based table is loaded. I think that could cause a problem (although I haven't tested your patch yet).
Yes, that was definitely a problem. I've included an incremental patch that shows the fixes to dm_init_request_based_queue below. I'll send out a revised patch: [PATCH 2/2 v2] ...
If you're comfortable with it please respond with your Acked-by.
Quoted 11 lines
Also, as you know, the table load can be canceled.
So initializing queue at table loading time may cause some weird behaviors for users. For example,# dmsetup create --notable bio-based # echo "0 100 multipath ..." | dmsetup load bio-based # echo "0 100 linear ..." | dmsetup load bio-based # dmsetup resume bio-based # ls /sys/block/dm-0/queue/ ... iosched ... If you (and Alasdair) think this behavior is acceptable, it might be OK. I just feel it's weird though...
Agreed, we don't think this is ideal.. I have a follow-on patch that I'll mail out separately for your consideration ([RFC PATCH 3/2]).
Quoted 10 lines
Quoted 4 lines
Having just looked at Nikanth's proposed DM patch 2/2 again it shows that
blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.Right, in general.
However, in this special case, I think initializing queue (allocating memory) during resume should be OK as I mentioned like below in:
http://marc.info/?l=linux-kernel&m=124999806420663&w=2
Quoted 7 lines
> Generally, dm must not allocate memory during resume because > it may cause a deadlock in no memory situation.
> However, there is no I/O on this device at this point, > so the allocation should be ok for this special case. > I think some comments are needed here to describe that.So you should be able to take this approach.
I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.
Here is the incremental patch I mentioned above.
---
Author: Mike Snitzer <[email protected]> Date: Wed May 12 18:52:01 2010 -0400
idempotent dm_init_request_based_queue
drivers/md/dm.c +19 -7
@@ -1951,14 +1951,30 @@ bad_module_get:return NULL ; } +/* +* Fully initialize the request_queue (elevator, ->request_fn, etc). +*/ int dm_init_request_based_queue (struct mapped_device *md) {struct request_queue *q =NULL ; - q =blk_init_allocated_queue (md->queue, dm_request_fn,NULL ); -if (!q) +if (!md->queue)return 0 ; - md->queue = q; + +/* +* Avoid re-initializing the queue (and leaking the existing +* elevator) if dm_init_request_based_queue() was already used. +*/ +if (!md->queue->elevator) { +/* Fully initialize the queue */ + q =blk_init_allocated_queue (md->queue, dm_request_fn,NULL ); +if (!q) +return 0 ; + md->queue = q; + md->saved_make_request_fn = md->queue->make_request_fn; +blk_queue_make_request (md->queue, dm_request); +elv_register_queue (md->queue); + }/* * Request-based dm devices cannot be stacked on top of bio-based dm
@@ -1970,7 +1986,6 @@ int dm_init_request_based_queue(struct mapped_device *md)* This queue is new, so no concurrency on the queue_flags. */ queue_flag_clear_unlocked (QUEUE_FLAG_STACKABLE , md->queue); - md->saved_make_request_fn = md->queue->make_request_fn;blk_queue_softirq_done (md->queue, dm_softirq_done);blk_queue_prep_rq (md->queue, dm_prep_fn);
@@ -1978,9 +1993,6 @@ int dm_init_request_based_queue(struct mapped_device *md)blk_queue_ordered (md->queue,QUEUE_ORDERED_DRAIN_FLUSH , dm_rq_prepare_flush); -/* Register the request-based queue's elevator with sysfs */ -elv_register_queue (md->queue); -return 1 ; }
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 23 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
Hi Mike,
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
Quoted 5 lines
Quoted 2 lines
By the way, your current patch has a problem below. It needs to be fixed at least.
Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).
I found another bug; blk_init_allocated_queue() overwrites q->unplug_fn with generic one. (Although it's currently harmless for multipath target.)
I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.
Quoted 3 lines
I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.
OK.
Then, how about the patch below?
It still fully initializes queue at device creation time for both bio-based and request-based. Then, it drops elevator when the device type is decided as bio-based at table binding time.
So no memory allocation during resume (freeing instead).
I hope this simple work-around approach is acceptable for you and Alasdair (and others).
(Then, let's focus on making a right fix rather than hacking
the block layer.)
[PATCH] dm: drop elevator when the device is bio-based
I/O scheduler affects nothing for bio-based dm device. Showing I/O scheduler's attributes in sysfs for bio-based dm devices is confusing users.
So drop them from sysfs when a dm device is decided as bio-based.
This patch depends on the commit below in the for-2.6.35 of Jens' linux-2.6-block git:
commit 01effb0dc1451fad55925873ffbfb88fa4eadce0
Author: Mike Snitzer <[email protected]> Date: Tue May 11 08:57:42 2010 +0200
block: allow initialization of previously allocated request_queue
--- drivers/md/dm.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/md/dm.c +8 -0
@@ -2410,6 +2410,14 @@ struct dm_table *dm_swap_table(struct ma goto out; } +/* drop elevator when the device type is decided as bio-based */ +if (!md->map &&dm_table_get_type (table) ==DM_TYPE_BIO_BASED ) { +elv_unregister_queue (md->queue); +elevator_exit (md->queue->elevator); + md->queue->request_fn =NULL ; + md->queue->elevator =NULL ; + } + map =__bind (md, table, &limits); out:
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 26 lines · Kiyoshi Ueda
Hi Mike,
Quoted 23 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
Hi Mike,
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
By the way, your current patch has a problem below. It needs to be fixed at least.
Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).
I found another bug;
blk_init_allocated_queue()overwritesq->unplug_fnwith generic one. (Although it's currently harmless for multipath target.)
Thanks again for your review!
Here is a simple incremental fix:
drivers/md/dm.c +13 -8
@@ -1854,6 +1854,17 @@ static const struct block_device_operations dm_blk_dops;static void dm_wq_work (struct work_struct *work);static void dm_rq_barrier_work (struct work_struct *work); +static void dm_init_md_queue (struct mapped_device *md) +{ + md->queue->queuedata = md; + md->queue->backing_dev_info.congested_fn = dm_any_congested; + md->queue->backing_dev_info.congested_data = md; +blk_queue_make_request (md->queue, dm_request); +blk_queue_bounce_limit (md->queue,BLK_BOUNCE_ANY ); + md->queue->unplug_fn = dm_unplug_all; +blk_queue_merge_bvec (md->queue, dm_merge_bvec); +} +/* * Allocate and initialise a blank device with a given minor. */
@@ -1895,13 +1906,7 @@ static struct mapped_device *alloc_dev(int minor)if (!md->queue) goto bad_queue; - md->queue->queuedata = md; - md->queue->backing_dev_info.congested_fn = dm_any_congested; - md->queue->backing_dev_info.congested_data = md; -blk_queue_make_request (md->queue, dm_request); -blk_queue_bounce_limit (md->queue,BLK_BOUNCE_ANY ); - md->queue->unplug_fn = dm_unplug_all; -blk_queue_merge_bvec (md->queue, dm_merge_bvec); +dm_init_md_queue (md); md->disk =alloc_disk (1 );if (!md->disk)
@@ -1974,7 +1979,7 @@ int dm_init_request_based_queue(struct mapped_device *md)return 0 ; md->queue = q; md->saved_make_request_fn = md->queue->make_request_fn; -blk_queue_make_request (md->queue, dm_request); +dm_init_md_queue (md); register_elevator =1 ; }else if (dm_bio_based_md (md)) {/*
Quoted 2 lines
I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.
The longer-term direction you want to take DM (known type during alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize the request_queue for a DM device (whether it is bio-based or request-based).
My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.
I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.
Quoted 8 lines
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.
Quoted 3 lines
I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.
Quoted 6 lines
OK.
Then, how about the patch below?
It still fully initializes queue at device creation time for both bio-based and request-based. Then, it drops elevator when the device type is decided as bio-based at table binding time.
So no memory allocation during resume (freeing instead).
I'm inclined to believe that the extra work for bio-based DM is not ideal. And even freeing during resume _could_ pose a problem during a resume (if it triggers unknown additional housekeeping). elv_unregister_queue() and elevator_exit() do quite a bit more than simply freeing memory. We can't safely assume those methods will never allocate memory to accomplish their cleanup (now or in the future).
Quoted 2 lines
I hope this simple work-around approach is acceptable for you and Alasdair (and others).
While it is certainly a small change it carries with it the disadvantage of still requiring more work than is needed for bio-based request_queue initialization. It also poses a risk by requiring additional work when resuming a bio-based device.
Quoted 2 lines
(Then, let's focus on making a right fix rather than hacking
the block layer.)
I haven't been hacking the block layer. Please don't misrepresent what I proposed.
Regards,
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 29 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Hi Mike,
Quoted 23 lines · Mike Snitzer
Quoted 11 lines · Kiyoshi Ueda
Hi Mike,
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
snip
Quoted 15 lines
Quoted 6 lines
I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.
Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.
The longer-term direction you want to take DM (known type during
alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize therequest_queuefor a DM device (whether it is bio-based or request-based).
Quoted 4 lines
My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.
Your patch makes that loading table affects the block device, so you are changing the model.
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.
My suggestion was (and is) to change the current model by determining bio-based/request-based at device creation time, not by a table.
Quoted 3 lines
I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.
The number of patches doesn't matter.
I still feel it's becoming complex, for example it now has to take care of reverting md/queue changes by previous preload.
Also, you have to take care of race condition between concurrent table loadings which was mentioned above, because the table which was used to initialized the queue may not be set in hc->new_map. Although such races may not happen in real usages, they could cause some critical problems (maybe kernel panic, maybe memory leak).
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 60 lines · Kiyoshi Ueda
Hi Mike,
Quoted 29 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Hi Mike,
Quoted 23 lines · Mike Snitzer
Quoted 11 lines · Kiyoshi Ueda
Hi Mike,
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
snip
Quoted 15 lines
Quoted 6 lines
I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.
Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.
The longer-term direction you want to take DM (known type during
alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize therequest_queuefor a DM device (whether it is bio-based or request-based).Quoted 4 lines
My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.
Quoted 8 lines
Your patch makes that loading table affects the block device, so you are changing the model.
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.
Nice catch, yes md->queue needs protection (see patch below).
Quoted 11 lines
My suggestion was (and is) to change the current model by determining bio-based/request-based at device creation time, not by a table.
Quoted 3 lines
I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.
The number of patches doesn't matter.
I still feel it's becoming complex, for example it now has to take care of reverting md/queue changes by previous preload.
AFAIK, no md/queue changes would need to be reverted. The last table load wins and the DM device's queue is left in the correct state for that winning table (assuming the patch at the end of this mail).
As for the overall complexity of my proposed changes, I'm inclined to address this issue without userspace changes. But I do understand we're at a cross-roads (and likely need Alasdair, and/or others, to weigh in):
1) Are my proposed kernel-only changes somehow detrimental to DM right now?
- We cannot let known bugs go unfixed because of unknown future requirements.
2) Is requiring the DM create ioctl to know that a device will be request-based vs bio-based really a reasonable requirement?
- To me a DM device's queue type is a DM (kernel) implementation detail that userspace shouldn't _ever_ need to know about. If userspace knew it'd make the kernel's life easier but that doesn't make it the correct solution on its own.
Quoted 5 lines
Also, you have to take care of race condition between concurrent table loadings which was mentioned above, because the table which was used to initialized the queue may not be set in
hc->new_map. Although such races may not happen in real usages, they could cause some critical problems (maybe kernel panic, maybe memory leak).
In the worst case, the hypothetical concurrent table loadings (of conflicting types: request vs bio) will just cause the request_queue to be fully initialized (allocated elevator) for bio-based devices. There aren't any other concerns with panics or memory leaks. Granted something like the following patch is required:
drivers/md/dm.c +20 -4
@@ -123,6 +123,11 @@ struct mapped_device {unsigned long flags;struct request_queue *queue; +/* +* Protect queue from concurrent initialization during +* table load +*/ +struct mutex queue_lock;struct gendisk *disk;char name[16 ];
@@ -1892,6 +1897,7 @@ static struct mapped_device *alloc_dev(int minor)init_rwsem (&md->io_lock);mutex_init (&md->suspend_lock); +mutex_init (&md->queue_lock);spin_lock_init (&md->deferred_lock);spin_lock_init (&md->barrier_error_lock);rwlock_init (&md->map_lock);
@@ -1966,14 +1972,16 @@ bad_module_get:*/ int dm_init_request_based_queue (struct mapped_device *md) { +int r =0 ;struct request_queue *q =NULL ; +mutex_lock (&md->queue_lock);/* Avoid re-initializing the queue if already fully initialized */ if (!md->queue->elevator) {/* Fully initialize the queue */ q =blk_init_allocated_queue (md->queue, dm_request_fn,NULL );if (!q) -return 0 ; + goto out; md->queue = q; md->saved_make_request_fn = md->queue->make_request_fn;dm_init_md_queue (md);
@@ -1984,8 +1992,11 @@ int dm_init_request_based_queue(struct mapped_device *md)* bio-based back to request-based, e.g.: rq -> bio -> rq */ md->queue->request_fn = dm_request_fn; - }else -return 1 ;/* already request-based */ + }else { +/* already request-based */ + r =1 ; + goto out; + }elv_register_queue (md->queue);
@@ -2006,7 +2017,10 @@ int dm_init_request_based_queue(struct mapped_device *md)blk_queue_ordered (md->queue,QUEUE_ORDERED_DRAIN_FLUSH , dm_rq_prepare_flush); -return 1 ; + r =1 ; +out: +mutex_unlock (&md->queue_lock); +return r; }void dm_clear_request_based_queue (struct mapped_device *md)
@@ -2015,8 +2029,10 @@ void dm_clear_request_based_queue(struct mapped_device *md)return ;/* already bio-based */ /* Unregister elevator from sysfs and clear ->request_fn */ +mutex_lock (&md->queue_lock);elv_unregister_queue (md->queue); md->queue->request_fn =NULL ; +mutex_unlock (&md->queue_lock); }static void unlock_fs (struct mapped_device *md);
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 54 lines · Mike Snitzer
Quoted 53 lines · Kiyoshi Ueda
Quoted 21 lines · Mike Snitzer
Quoted 20 lines · Kiyoshi Ueda
Quoted 19 lines · Mike Snitzer
Quoted 9 lines · Kiyoshi Ueda
Quoted 7 lines · Mike Snitzer
It is clear we need to resolve the current full
request_queueinitialization that occurs even for bio-based DM devices.I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.
Quoted 5 lines
Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.
snip
Quoted 15 lines
Quoted 6 lines
I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.
Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.
The longer-term direction you want to take DM (known type during
alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize therequest_queuefor a DM device (whether it is bio-based or request-based).Quoted 4 lines
My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.Your patch makes that loading table affects the block device, so you are changing the model.
Quoted 7 lines
Quoted 5 lines
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.Nice catch, yes
md->queueneeds protection (see patch below).
Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of the hash_cell for the mapped_device in protection by _hash_lock.
Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the mapped_device while the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 28 lines · Kiyoshi Ueda
Hi Mike,
Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.Your patch makes that loading table affects the block device, so you are changing the model.
Quoted 7 lines
Quoted 5 lines
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.Nice catch, yes
md->queueneeds protection (see patch below).Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of thehash_cellfor themapped_devicein protection by_hash_lock.
Quoted 4 lines
Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the
mapped_devicewhile the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.
Ah, yes.. I looked at the possibility of serializing the entirety of table_load but determined that would be too excessive (would reduce parallelism of table_load). But I clearly missed the fact that there could be a race to the _hash_lock protected critical section in table_load() -- leading to queue inconsistency.
I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load(). So it sets the queue up _after_ the table's type is established with dm_table_set_type().
Thanks again for your review; I'll work to be more meticulous in the future!
drivers/md/dm-ioctl.c +8 -0
@@ -1180,6 +1180,14 @@ static int table_load(struct dm_ioctl *param, size_t param_size) goto out; } + r =dm_table_setup_md_queue (t); +if (r) { +DMWARN ("unable to setup device queue for this table" ); +dm_table_destroy (t); +up_write (&_hash_lock); + goto out; + } +if (hc->new_map)dm_table_destroy (hc->new_map); hc->new_map = t;
drivers/md/dm-table.c +17 -6
@@ -803,7 +803,6 @@ int dm_table_set_type(struct dm_table *t)if (bio_based) {/* We must use this table as bio-based */ t->type =DM_TYPE_BIO_BASED ; -dm_clear_request_based_queue (t->md);return 0 ; }
@@ -830,11 +829,6 @@ int dm_table_set_type(struct dm_table *t)return -EINVAL ; } -if (!dm_init_request_based_queue (t->md)) { -DMWARN ("Cannot initialize queue for Request-based dm" ); -return -EINVAL ; - } - t->type =DM_TYPE_REQUEST_BASED ;return 0 ;
@@ -850,6 +844,23 @@ bool dm_table_request_based(struct dm_table *t)return dm_table_get_type (t) ==DM_TYPE_REQUEST_BASED ; } +/* +* Setup the DM device's queue based on table's type. +* Caller must serialize access to table's md. +*/ +int dm_table_setup_md_queue (struct dm_table *t) +{ +if (dm_table_request_based (t)) { +if (!dm_init_request_based_queue (t->md)) { +DMWARN ("Cannot initialize queue for Request-based dm" ); +return -EINVAL ; + } + }else +dm_clear_request_based_queue (t->md); + +return 0 ; +} +int dm_table_alloc_md_mempools (struct dm_table *t) {unsigned type =dm_table_get_type (t);
drivers/md/dm.c +11 -11
@@ -1856,6 +1856,17 @@ static void dm_rq_barrier_work(struct work_struct *work);static void dm_init_md_queue (struct mapped_device *md) { +/* +* Request-based dm devices cannot be stacked on top of bio-based dm +* devices. The type of this dm device has not been decided yet. +* The type is decided at the first table loading time. +* To prevent problematic device stacking, clear the queue flag +* for request stacking support until then. +* +* This queue is new, so no concurrency on the queue_flags. +*/ +queue_flag_clear_unlocked (QUEUE_FLAG_STACKABLE , md->queue); + md->queue->queuedata = md; md->queue->backing_dev_info.congested_fn = dm_any_congested; md->queue->backing_dev_info.congested_data = md;
@@ -1989,17 +2000,6 @@ int dm_init_request_based_queue(struct mapped_device *md)elv_register_queue (md->queue); -/* -* Request-based dm devices cannot be stacked on top of bio-based dm -* devices. The type of this dm device has not been decided yet. -* The type is decided at the first table loading time. -* To prevent problematic device stacking, clear the queue flag -* for request stacking support until then. -* -* This queue is new, so no concurrency on the queue_flags. -*/ -queue_flag_clear_unlocked (QUEUE_FLAG_STACKABLE , md->queue); -blk_queue_softirq_done (md->queue, dm_softirq_done);blk_queue_prep_rq (md->queue, dm_prep_fn);blk_queue_lld_busy (md->queue, dm_lld_busy);
drivers/md/dm.h +1 -0
@@ -65,6 +65,7 @@ bool dm_table_request_based(struct dm_table *t);int dm_table_alloc_md_mempools (struct dm_table *t);void dm_table_free_md_mempools (struct dm_table *t);struct dm_md_mempools *dm_table_get_md_mempools (struct dm_table *t); +int dm_table_setup_md_queue (struct dm_table *t);/* * To check the return value from dm_table_find_target().
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 38 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.Your patch makes that loading table affects the block device, so you are changing the model.
Quoted 7 lines
Quoted 5 lines
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.Nice catch, yes
md->queueneeds protection (see patch below).Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of thehash_cellfor themapped_devicein protection by_hash_lock.Quoted 4 lines
Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the
mapped_devicewhile the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.Ah, yes.. I looked at the possibility of serializing the entirety of
table_loadbut determined that would be too excessive (would reduce parallelism oftable_load). But I clearly missed the fact that there could be a race to the_hash_lockprotected critical section intable_load()-- leading to queue inconsistency.
Quoted 6 lines
I'll post v5 of the overall patch which will revert the
mapped_device'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the_hash_lockprotected critical section intable_load(). So it sets the queue up _after_ the table's type is established withdm_table_set_type().
dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 48 lines · Kiyoshi Ueda
Hi Mike,
Quoted 38 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume If we want to fix a problem, I think we should either work based on this model or change the model.Your patch makes that loading table affects the block device, so you are changing the model.
Quoted 7 lines
Quoted 5 lines
If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.Nice catch, yes
md->queueneeds protection (see patch below).Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of thehash_cellfor themapped_devicein protection by_hash_lock.Quoted 4 lines
Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the
mapped_devicewhile the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.Ah, yes.. I looked at the possibility of serializing the entirety of
table_loadbut determined that would be too excessive (would reduce parallelism oftable_load). But I clearly missed the fact that there could be a race to the_hash_lockprotected critical section intable_load()-- leading to queue inconsistency.Quoted 6 lines
I'll post v5 of the overall patch which will revert the
mapped_device'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the_hash_lockprotected critical section intable_load(). So it sets the queue up _after_ the table's type is established withdm_table_set_type().
Quoted 4 lines
dm_table_setup_md_queue()may allocate memory with blocking mode. Blocking allocation inside exclusive_hash_lockcan cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.
We make no guarantees that other DM devices will resume before a table load -- so calling dm_table_setup_md_queue() within the exclusive _hash_lock is no different than other DM devices being suspended while a request-based DM device performs its first table_load().
My thinking was this should not be a problem as it is only valid to call dm_table_setup_md_queue() before the newly created request-based DM device has been resumed.
AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any GFP_KERNEL allocation could recurse (elevator_alloc() currently uses GFP_KERNEL with kmalloc_node)...
I'll have to review the DM code further to see if all memory allocations during table_load() are done via mempools. I'll also bring this up on this week's LVM call.
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.
Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */
This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.
Thanks,
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 25 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda
Hi Mike,
Quoted 6 lines · Mike Snitzer
I'll post v5 of the overall patch which will revert the
mapped_device'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the_hash_lockprotected critical section intable_load(). So it sets the queue up _after_ the table's type is established withdm_table_set_type().Quoted 4 lines
dm_table_setup_md_queue()may allocate memory with blocking mode. Blocking allocation inside exclusive_hash_lockcan cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.We make no guarantees that other DM devices will resume before a table load -- so calling
dm_table_setup_md_queue()within the exclusive_hash_lockis no different than other DM devices being suspended while a request-based DM device performs its firsttable_load().My thinking was this should not be a problem as it is only valid to call
dm_table_setup_md_queue()before the newly created request-based DM device has been resumed.
Quoted 8 lines
AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any
GFP_KERNELallocation could recurse (elevator_alloc()currently usesGFP_KERNELwithkmalloc_node)...I'll have to review the DM code further to see if all memory allocations during
table_load()are done via mempools. I'll also bring this up on this week's LVM call.
We discussed this and I understand the scope of the problem now.
Just reiterating what you covered when you first pointed this issue out:
It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive _hash_lock while blocking. Having _hash_lock prevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.
I hope to have v6 available at some point today but it may be delayed by a day.
Quoted 5 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.
Quoted 8 lines
Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.
I'll look to address this issue in v6 too.
Regards,
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 7 lines · Mike Snitzer
Quoted 7 lines · Kiyoshi Ueda
Quoted 6 lines
Hi Mike,
dm_table_setup_md_queue()may allocate memory with blocking mode. Blocking allocation inside exclusive_hash_lockcan cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.
<snip>
Quoted 11 lines
We discussed this and I understand the scope of the problem now.
Just reiterating what you covered when you first pointed this issue out:
It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive
_hash_lockwhile blocking. Having_hash_lockprevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.
Quoted 3 lines
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But
s/multiple_device/mapped_device/
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
In reply to Mike SnitzerHi Mike,
Quoted 35 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Kiyoshi Ueda
Quoted 6 lines · Mike Snitzer
I'll post v5 of the overall patch which will revert the
mapped_device'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the_hash_lockprotected critical section intable_load(). So it sets the queue up _after_ the table's type is established withdm_table_set_type().Quoted 4 lines
dm_table_setup_md_queue()may allocate memory with blocking mode. Blocking allocation inside exclusive_hash_lockcan cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.We make no guarantees that other DM devices will resume before a table load -- so calling
dm_table_setup_md_queue()within the exclusive_hash_lockis no different than other DM devices being suspended while a request-based DM device performs its firsttable_load().My thinking was this should not be a problem as it is only valid to call
dm_table_setup_md_queue()before the newly created request-based DM device has been resumed.Quoted 8 lines
AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any
GFP_KERNELallocation could recurse (elevator_alloc()currently usesGFP_KERNELwithkmalloc_node)...I'll have to review the DM code further to see if all memory allocations during
table_load()are done via mempools. I'll also bring this up on this week's LVM call.We discussed this and I understand the scope of the problem now.
Quoted 8 lines
Just reiterating what you covered when you first pointed this issue out:
It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive
_hash_lockwhile blocking. Having_hash_lockprevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.
That's right.
Quoted 4 lines
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking_hash_lock.
You have to see do_resume(), too.do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)
dm_clear_request_based_queue(md)
md->queue->request_fn = NULL
down_write(_hash_lock)
new_map = hc->new_map (<= rq-based)
up_write(_hash_lock)
dm_swap_table(md, new_map)
__bind(md, new_map) (<= rq-based) down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)
Other ioctls might be as well, since I have checked only interaction between table load and resume.
Quoted 2 lines
I hope to have v6 available at some point today but it may be delayed by a day.
Sorry for repeating but since you are trying to change the current dm model, please design/consider it carefully; what objects/ioctls are involved and how you change their relationships, where are the critical sections and how you protect them.
Otherwise, I'm afraid we will just spend your time in transforming one bug to another. (and my time for reviewing and verifying problems :)
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
Quoted 7 lines
Quoted 5 lines
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.I'll look to address this issue in v6 too.
It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)
dm_clear_request_based_queue(md)
md->queue->request_fn = NULLThanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 22 lines · Kiyoshi Ueda
Hi Mike,
Quoted 4 lines · Mike Snitzer
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking_hash_lock.You have to see
do_resume(), too.do_resume()getshc->new_mapwith_hash_lockheld but without your 'queue_lock' held. This could cause inconsistent status;
table_load()for bio-based tabledo_resume()for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t)(<= bio-based)dm_clear_request_based_queue(md) md->queue->request_fn = NULL down_write(_hash_lock) new_map = hc->new_map (<= rq-based) up_write(_hash_lock) dm_swap_table(md, new_map) __bind(md, new_map) (<= rq-based)
Quoted 4 lines
down_write(_hash_lock)
hc->new_map= t
up_write(_hash_lock)
dm_unlock_md_queue(md)
Understood.
Quoted 2 lines
Other ioctls might be as well, since I have checked only interaction between table load and resume.
Table load and resume should be the only ioctls that are relevant.
Quoted 9 lines
Quoted 2 lines
I hope to have v6 available at some point today but it may be delayed by a day.
Sorry for repeating but since you are trying to change the current dm model, please design/consider it carefully; what objects/ioctls are involved and how you change their relationships, where are the critical sections and how you protect them.
Otherwise, I'm afraid we will just spend your time in transforming one bug to another. (and my time for reviewing and verifying problems :)
Noted. I do appreciate your thorough reviews but will hopefully reduce your burden by improving my awareness and code.
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
Quoted 7 lines
Quoted 5 lines
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.I'll look to address this issue in v6 too.
It can race between
table_load()anddo_resume()as well;
table_load()for bio-baseddo_resume()for rq-based
---------------------------------------------------------------------
dm_table_set_type(t)(<= bio-based)
live_table=dm_get_live_table(md)
(assume nolive_tableyet)
new_map=hc->new_map(<= rq-based)dm_swap_table(md, new_map)__bind(md, new_map)
dm_table_setup_md_queue(t)
Quoted 2 lines
dm_clear_request_based_queue(md)
md->queue->request_fn = NULLUnderstood.
I've addressed the above races by:
1) properly protecting md->queue during do_resume(). Both do_resume() and table_load() first take md->queue_lock and then _hash_lock.
2) adding a negative check for an existing live table to dm_table_setup_md_queue().
I will mail out v7 after I've given it additional review.
FYI, I've revised and split out the conflicting table type checking patch as that patch should not be controversial and is completely independent of this DM queue initialization work. I'll email it to dm-devel soon.
Regards,
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 25 lines · Mike Snitzer
Quoted 24 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking_hash_lock.You have to see
do_resume(), too.do_resume()getshc->new_mapwith_hash_lockheld but without your 'queue_lock' held. This could cause inconsistent status;
table_load()for bio-based tabledo_resume()for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t)(<= bio-based)dm_clear_request_based_queue(md) md->queue->request_fn = NULL down_write(_hash_lock) new_map = hc->new_map (<= rq-based) up_write(_hash_lock) dm_swap_table(md, new_map) __bind(md, new_map) (<= rq-based)
down_write(_hash_lock)
hc->new_map= t
up_write(_hash_lock)
dm_unlock_md_queue(md)
snip
Quoted 31 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
Quoted 7 lines
Quoted 5 lines
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.I'll look to address this issue in v6 too.
It can race between
table_load()anddo_resume()as well;
table_load()for bio-baseddo_resume()for rq-based
---------------------------------------------------------------------
dm_table_set_type(t)(<= bio-based)
live_table=dm_get_live_table(md)
(assume nolive_tableyet)
new_map=hc->new_map(<= rq-based)dm_swap_table(md, new_map)__bind(md, new_map)
dm_table_setup_md_queue(t)Quoted 2 lines
dm_clear_request_based_queue(md) md->queue->request_fn = NULLUnderstood.
Quoted 7 lines
I've addressed the above races by:
1) properly protectingmd->queueduringdo_resume(). Bothdo_resume()andtable_load()first takemd->queue_lockand then_hash_lock.
2) adding a negative check for an existing live table todm_table_setup_md_queue().I will mail out v7 after I've given it additional review.
dm_table_setup_md_queue() may allocate memory with blocking mode inside md->queue_lock which is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded in hc->new_map and no live table yet in a mapped_device.
- Load a request-based table and then no memory situation happens in dm_table_setup_md_queue().
- Then, the mapped_device can't be resumed even in the case that resuming the mapped_device with the preloaded bio-based table can make some memory by writeback.
=> deadlock
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Quoted 68 lines · Kiyoshi Ueda
Hi Mike,
Quoted 25 lines · Mike Snitzer
Quoted 24 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking_hash_lock.You have to see
do_resume(), too.do_resume()getshc->new_mapwith_hash_lockheld but without your 'queue_lock' held. This could cause inconsistent status;
table_load()for bio-based tabledo_resume()for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t)(<= bio-based)dm_clear_request_based_queue(md) md->queue->request_fn = NULL down_write(_hash_lock) new_map = hc->new_map (<= rq-based) up_write(_hash_lock) dm_swap_table(md, new_map) __bind(md, new_map) (<= rq-based)
down_write(_hash_lock)
hc->new_map= t
up_write(_hash_lock)
dm_unlock_md_queue(md)snip
Quoted 31 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
Quoted 7 lines
Quoted 5 lines
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.I'll look to address this issue in v6 too.
It can race between
table_load()anddo_resume()as well;
table_load()for bio-baseddo_resume()for rq-based
---------------------------------------------------------------------
dm_table_set_type(t)(<= bio-based)
live_table=dm_get_live_table(md)
(assume nolive_tableyet)
new_map=hc->new_map(<= rq-based)dm_swap_table(md, new_map)__bind(md, new_map)
dm_table_setup_md_queue(t)Quoted 2 lines
dm_clear_request_based_queue(md) md->queue->request_fn = NULLUnderstood.
Quoted 7 lines
I've addressed the above races by:
1) properly protectingmd->queueduringdo_resume(). Bothdo_resume()andtable_load()first takemd->queue_lockand then_hash_lock.
2) adding a negative check for an existing live table todm_table_setup_md_queue().I will mail out v7 after I've given it additional review.
Quoted 12 lines
dm_table_setup_md_queue()may allocate memory with blocking mode insidemd->queue_lockwhich is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded inhc->new_mapand no live table yet in amapped_device.
- Load a request-based table and then no memory situation happens indm_table_setup_md_queue().
- Then, themapped_devicecan't be resumed even in the case that resuming themapped_devicewith the preloaded bio-based table can make some memory by writeback.
=> deadlock
My patch introduces a new mutex that is local to the mapped_device being loaded. And that same mutex is taken during do_resume (just like you implicitly said was needed to avoid races).
But I'm _very_ skeptical of the validity of this deadlock scenario. You're leaving out some important context for why we might expect a DM device to service (queue) IO during the table load _before_ the device has _ever_ been resumed. When I issue IO to a DM device that only has an inactive table it returns immediately:
# dmsetup table --inactive bio-based: 0 8388608 linear 254:16 384 # dd if=/dev/mapper/bio-based of=/dev/null bs=1024k count=10 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00016418 s, 0.0 kB/s
As we discussed earlier in this thread, a table load can block indefinitely (waiting for memory) as long as _hash_lock is not held (so other DM ioctls are still possible).
It is a serious reach to suggest that your new deadlock scenario is a legitimate concern. This would mean that a user would:
1) mistakenly load a bio-based table when they meant rq-based
2) load rq-based -- but block waiting for elevator_alloc()
3) attempt to resume device with bio-based anyway -- to make forward progress with writeback destined to the device being resumed for the first time? -- but again, why would dirty pages be destined for this inactive device already?
4) even if the user could resume they'd be left with a bio-based device; which isn't what they wanted..
Do you see the obscure nature of the scenario you've presented?
Please help me understand what I am missing?
Mike
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
Hi Mike,
Quoted 75 lines · Mike Snitzer
Quoted 61 lines · Kiyoshi Ueda
Quoted 24 lines · Mike Snitzer
Quoted 23 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer
Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the
multiple_device(effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking_hash_lock.You have to see
do_resume(), too.do_resume()getshc->new_mapwith_hash_lockheld but without your 'queue_lock' held. This could cause inconsistent status;
table_load()for bio-based tabledo_resume()for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t)(<= bio-based)dm_clear_request_based_queue(md) md->queue->request_fn = NULL down_write(_hash_lock) new_map = hc->new_map (<= rq-based) up_write(_hash_lock) dm_swap_table(md, new_map) __bind(md, new_map) (<= rq-based)
down_write(_hash_lock)
hc->new_map= t
up_write(_hash_lock)
dm_unlock_md_queue(md)snip
Quoted 27 lines
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
Quoted 6 lines
Quoted 5 lines
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.I'll look to address this issue in v6 too.
It can race between
table_load()anddo_resume()as well;
table_load()for bio-baseddo_resume()for rq-based
---------------------------------------------------------------------
dm_table_set_type(t)(<= bio-based)
live_table=dm_get_live_table(md)
(assume nolive_tableyet)
new_map=hc->new_map(<= rq-based)dm_swap_table(md, new_map)__bind(md, new_map)
dm_table_setup_md_queue(t)Quoted 2 lines
dm_clear_request_based_queue(md) md->queue->request_fn = NULLUnderstood.
Quoted 7 lines
I've addressed the above races by:
1) properly protectingmd->queueduringdo_resume(). Bothdo_resume()andtable_load()first takemd->queue_lockand then_hash_lock.
2) adding a negative check for an existing live table todm_table_setup_md_queue().I will mail out v7 after I've given it additional review.
Quoted 12 lines
dm_table_setup_md_queue()may allocate memory with blocking mode insidemd->queue_lockwhich is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded inhc->new_mapand no live table yet in amapped_device.
- Load a request-based table and then no memory situation happens indm_table_setup_md_queue().
- Then, themapped_devicecan't be resumed even in the case that resuming themapped_devicewith the preloaded bio-based table can make some memory by writeback.
=> deadlock
Quoted 8 lines
My patch introduces a new mutex that is local to the
mapped_devicebeing loaded. And that same mutex is taken duringdo_resume(just like you implicitly said was needed to avoid races).But I'm _very_ skeptical of the validity of this deadlock scenario. You're leaving out some important context for why we might expect a DM device to service (queue) IO during the table load _before_ the device has _ever_ been resumed.
Because upper device might wait for the DM device's resume to serve pending I/O.
In a stacked configuration, load/suspend/resume happens in this order:
1. load tables to upper and lower(s)
2. suspend upper
3. suspend lower(s)
4. resume lower(s)
5. resume upper
And "lower(s)" may include a new DM device which has never been resumed before the step 4.
So, the following example could cause the deadlock scenario:
# echo "0 10 linear /dev/sda 0" | dmsetup create lower0 # echo "0 10 linear /dev/mapper/lower0 0" | dmsetup create upper # dmsetup create --notable lower1 # echo "0 10 linear /dev/mapper/lower1 0" | dmsetup load upper # echo "0 10 linear /dev/sdb 0" dmsetup load lower1 <Any I/Os to /dev/mapper/upper is possible here> # dmsetup suspend upper <I/Os are queued on /dev/mapper/upper> # echo "0 10 multipath ../dev/sda../dev/sdb.." | dmsetup load lower1
If no memory happens here, the system may be hanged up because /dev/mapper/lower1 can't be resumed where /dev/mapper/lower1 will be needed for writeback for /dev/mapper/upper when /dev/mapper/upper is resumed.
(Without your patch, /dev/mapper/upper and /dev/mapper/lower1 could
be resumed using pre-loaded bio-based tables and the system could get some memory and be alive.)
Quoted 7 lines
As we discussed earlier in this thread, a table load can block indefinitely (waiting for memory) as long as
_hash_lockis not held (so other DM ioctls are still possible).It is a serious reach to suggest that your new deadlock scenario is a legitimate concern. This would mean that a user would:
Quoted 8 lines
1) mistakenly load a bio-based table when they meant rq-based
2) load rq-based -- but block waiting forelevator_alloc()
3) attempt to resume device with bio-based anyway -- to make forward progress with writeback destined to the device being resumed for the first time? -- but again, why would dirty pages be destined for this inactive device already?
4) even if the user could resume they'd be left with a bio-based device; which isn't what they wanted..
Although the scenario/example which I mentioned above may be too intentional, it is still possible.
We should design not to cause such critical problems or prevent/reject such problematic operations.
Your new model, which determines md type at the first table loading time and prevents type switching, is close to my initial idea, which determines md type at device creation time, and no userspace change is required. I think it's reasonable model.
But I have some comments on that, so I'll reply to the new thread of your new model patch-set.
Thanks,
Kiyoshi Ueda
Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device
In reply to Mike SnitzerQuoted 6 lines · Mike Snitzer
Quoted 4 lines · Kiyoshi Ueda
Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a
mapped_devicewhich is already active/used as request-based setsq->requst_fnin NULL.) That could cause some critical problems.
Quoted 8 lines
Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.
dm_swap_table()has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */This check should come during
table_load, as part ofdm_table_set_type(), rather than during table resume.
FYI, the following patch is the relevant portion of the v6 patch that addresses the issue discussed above.
drivers/md/dm-table.c +14 -1
@@ -782,6 +782,7 @@ int dm_table_set_type(struct dm_table *t) {unsigned i;unsigned bio_based =0 , request_based =0 ; +struct dm_table *live_table =NULL ;struct dm_target *tgt;struct dm_dev_internal *dd;struct list_head *devices;
@@ -803,7 +804,7 @@ int dm_table_set_type(struct dm_table *t)if (bio_based) {/* We must use this table as bio-based */ t->type =DM_TYPE_BIO_BASED ; -return 0 ; + goto finish; }BUG_ON (!request_based);/* No targets in this table */
@@ -831,6 +832,18 @@ int dm_table_set_type(struct dm_table *t) t->type =DM_TYPE_REQUEST_BASED ; +finish: +/* cannot change the device type, once a table is bound */ + live_table =dm_get_live_table (t->md); +if (live_table) { +if (dm_table_get_type (live_table) !=dm_table_get_type (t)) { +DMWARN ("can't change the device type after a table is bound" ); +dm_table_put (live_table); +return -EINVAL ; + } +dm_table_put (live_table); + } +return 0 ; }
drivers/md/dm.c +0 -7
@@ -2470,13 +2470,6 @@ struct dm_table *dm_swap_table(struct mapped_device *md, struct dm_table *table) goto out; } -/* cannot change the device type, once a table is bound */ -if (md->map && - (dm_table_get_type (md->map) !=dm_table_get_type (table))) { -DMWARN ("can't change the device type after a table is bound" ); - goto out; - } - map =__bind (md, table, &limits); out:
PATCH v2 2/2 dm: only initialize full request_queue for request-based device
In reply to Mike SnitzerRevert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.
dm_init_request_based_queue is now called at the end of a request-based DM device's table load.
Otherwise bio-based DM devices will have an elevator, request_fn, etc. As a result bio-based DM devices had an 'iosched/' tree as well as a 'scheduler' other than "none" in sysfs.
--- drivers/md/dm-table.c | 5 ++++ drivers/md/dm.c | 64 +++++++++++++++++++++++++++++++++++------------- drivers/md/dm.h | 2 + 3 files changed, 53 insertions(+), 18 deletions(-)
drivers/md/dm-table.c +5 -0
@@ -829,6 +829,11 @@ int dm_table_set_type(struct dm_table *t)return -EINVAL ; } +if (!dm_init_request_based_queue (t->md)) { +DMWARN ("Cannot initialize queue for Request-based dm" ); +return -EINVAL ; + } + t->type =DM_TYPE_REQUEST_BASED ;return 0 ;
drivers/md/dm.c +46 -18
85 lines not on this page · raw message
drivers/md/dm.h +2 -0
@@ -113,6 +113,8 @@ void dm_sysfs_exit(struct mapped_device *md);struct kobject *dm_kobject (struct mapped_device *md);struct mapped_device *dm_get_from_kobject (struct kobject *kobj); +int dm_init_request_based_queue (struct mapped_device *md); +/* * Targets for linear and striped mappings */
[RFC PATCH 3/2] dm: bio-based device must not register elevator in sysfs
Properly clear io scheduler state from sysfs if transitioning from a request-based table back to bio-based. Unregister the elevator from sysfs (removes 'iosched/') and have the 'scheduler' sysfs attribute properly report "none" for bio-based DM.
Adjust elv_iosched_show() to return "none" if !blk_queue_stackable. This allows the block layer to take bio-based DM into consideration.
These changes address the following (albeit obscure) corner-case:
# dmsetup create --notable bio-based # echo "0 100 multipath ..." | dmsetup load bio-based # echo "0 100 linear ..." | dmsetup load bio-based # dmsetup resume bio-based # ls /sys/block/dm-0/queue/ ... iosched ...
--- block/elevator.c | 2 +- drivers/md/dm-table.c | 1 + drivers/md/dm.c | 41 ++++++++++++++++++++++++++++++++++++++--- drivers/md/dm.h | 1 + 4 files changed, 41 insertions(+), 4 deletions(-)
block/elevator.c +1 -1
@@ -1088,7 +1088,7 @@ ssize_t elv_iosched_show(struct request_queue *q, char *name)struct elevator_type *__e;int len =0 ; -if (!q->elevator) +if (!q->elevator || !blk_queue_stackable (q))return sprintf (name,"none\n" ); elv = e->elevator_type;
drivers/md/dm-table.c +1 -0
@@ -803,6 +803,7 @@ int dm_table_set_type(struct dm_table *t)if (bio_based) {/* We must use this table as bio-based */ t->type =DM_TYPE_BIO_BASED ; +dm_clear_request_based_queue (t->md);return 0 ; }
drivers/md/dm.c +38 -3
62 lines not on this page · raw message
drivers/md/dm.h +1 -0
@@ -114,6 +114,7 @@ struct kobject *dm_kobject(struct mapped_device *md);struct mapped_device *dm_get_from_kobject (struct kobject *kobj);int dm_init_request_based_queue (struct mapped_device *md); +void dm_clear_request_based_queue (struct mapped_device *md);/* * Targets for linear and striped mappings
[PATCH 3/2 v2] dm: bio-based device must not register elevator in sysfs
Properly clear io scheduler state from sysfs if transitioning from a request-based table back to bio-based. Unregister the elevator from sysfs (removes 'iosched/') and have the 'scheduler' sysfs attribute properly report "none" for bio-based DM.
Adjust elv_iosched_show() to return "none" if !blk_queue_stackable. This allows the block layer to take bio-based DM into consideration.
These changes address the following (albeit obscure) corner-case:
# dmsetup create --notable bio-based # echo "0 100 multipath ..." | dmsetup load bio-based # echo "0 100 linear ..." | dmsetup load bio-based # dmsetup resume bio-based # ls /sys/block/dm-0/queue/ ... iosched ...
--- block/elevator.c | 2 +- drivers/md/dm-table.c | 1 + drivers/md/dm.c | 44 ++++++++++++++++++++++++++++++++++++-------- drivers/md/dm.h | 1 + 4 files changed, 39 insertions(+), 9 deletions(-)
v2 changes:
* switched to using BUG_ON in dm_{init,clear}_request_based_queue
* refined dm_clear_request_based_queue slightly
* introduced dm_bio_based_md
* cleaned up some comments
drivers/md/dm-table.c +1 -0
@@ -803,6 +803,7 @@ int dm_table_set_type(struct dm_table *tif (bio_based) {/* We must use this table as bio-based */ t->type =DM_TYPE_BIO_BASED ; +dm_clear_request_based_queue (t->md);return 0 ; }
drivers/md/dm.c +36 -8
76 lines not on this page · raw message
drivers/md/dm.h +1 -0
@@ -114,6 +114,7 @@ struct kobject *dm_kobject(struct mappedstruct mapped_device *dm_get_from_kobject (struct kobject *kobj);int dm_init_request_based_queue (struct mapped_device *md); +void dm_clear_request_based_queue (struct mapped_device *md);/* * Targets for linear and striped mappings
block/elevator.c +1 -1
@@ -1086,7 +1086,7 @@ ssize_t elv_iosched_show(struct request_struct elevator_type *__e;int len =0 ; -if (!q->elevator) +if (!q->elevator || !blk_queue_stackable (q))return sprintf (name,"none\n" ); elv = e->elevator_type;
Quoted 7 lines · Mike Snitzer
blk_init_queue()allocates therequest_queuestructure and then initializes it as needed (request_fn, elevator, etc).Split initialization out to
blk_init_allocated_queue_node. Introduceblk_init_allocated_queuewrapper function to model existingblk_init_queueandblk_init_queue_nodeinterfaces.
Quoted 7 lines
Export
elv_register_queueto allow a newly added elevator to be registered with sysfs. Exportelv_unregister_queuefor symmetry.These changes allow DM to initialize a device's
request_queuewith more precision. In particular, DM no longer unconditionally initializes a fullrequest_queue(elevator et al). It only does so for a request-based DM device.
Looks good, I'll apply this. I don't think the exports need to be _GPL in this case, generally I've only done that with the exports for hooking in a new IO scheduler.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Mike Snitzer
blk_init_queue()allocates therequest_queuestructure and then initializes it as needed (request_fn, elevator, etc).Split initialization out to
blk_init_allocated_queue_node. Introduceblk_init_allocated_queuewrapper function to model existingblk_init_queueandblk_init_queue_nodeinterfaces.Export
elv_register_queueto allow a newly added elevator to be registered with sysfs. Exportelv_unregister_queuefor symmetry.
Quoted 8 lines
Quoted 4 lines
These changes allow DM to initialize a device's
request_queuewith more precision. In particular, DM no longer unconditionally initializes a fullrequest_queue(elevator et al). It only does so for a request-based DM device.Looks good, I'll apply this. I don't think the exports need to be
_GPLin this case, generally I've only done that with the exports for hooking in a new IO scheduler.
OK, thanks for picking this up. Do you want me to send v2 that drops the _GPL or will you make those changes?
Mike
Quoted 19 lines · Mike Snitzer
Quoted 16 lines · Jens Axboe
Quoted 14 lines · Mike Snitzer
blk_init_queue()allocates therequest_queuestructure and then initializes it as needed (request_fn, elevator, etc).Split initialization out to
blk_init_allocated_queue_node. Introduceblk_init_allocated_queuewrapper function to model existingblk_init_queueandblk_init_queue_nodeinterfaces.Export
elv_register_queueto allow a newly added elevator to be registered with sysfs. Exportelv_unregister_queuefor symmetry.These changes allow DM to initialize a device's
request_queuewith more precision. In particular, DM no longer unconditionally initializes a fullrequest_queue(elevator et al). It only does so for a request-based DM device.
Quoted 6 lines
Quoted 3 lines
Looks good, I'll apply this. I don't think the exports need to be
_GPLin this case, generally I've only done that with the exports for hooking in a new IO scheduler.OK, thanks for picking this up. Do you want me to send v2 that drops the
_GPLor will you make those changes?
I just hand-edited that part. I can add the dm bit as well, when it has been reviewed and acked.
Signature · 1 line
Jens Axboe