Skip to contentSkip to conversation

block: allow initialization of previously allocated request_queue

RFC PATCH In review Mike Snitzer 3 participants 28 messages May 10, 2010 – May 24, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/2 block: allow initialization of previously allocated request_queue

blk_init_queue() allocates the request_queue structure and then initializes it as needed (request_fn, elevator, etc).

Split initialization out to blk_init_allocated_queue_node. Introduce blk_init_allocated_queue wrapper function to model existing blk_init_queue and blk_init_queue_node interfaces.

Export elv_register_queue to allow a newly added elevator to be registered with sysfs. Export elv_unregister_queue for symmetry.

These changes allow DM to initialize a device's request_queue with more precision. In particular, DM no longer unconditionally initializes a full request_queue (elevator et al). It only does so for a request-based DM device.

---
 block/blk-core.c       |   18 +++++++++++++++++-
 block/elevator.c       |    2 ++
 include/linux/blkdev.h |    5 +++++
 3 files changed, 24 insertions(+), 1 deletions(-)
block/blk-core.c +17 -1
index 9fe174d..bfd420c 100644
@@ -567,7 +567,23 @@ struct request_queue *
 blk_init_queue_node(request_fn_proc *rfn, spinlock_t *lock, int node_id)
 {
 	struct request_queue *q = blk_alloc_queue_node(GFP_KERNEL, node_id);
+	
+	return blk_init_allocated_queue_node(q, rfn, lock, node_id);
+}
+EXPORT_SYMBOL(blk_init_queue_node);
 
+struct request_queue *
+blk_init_allocated_queue(struct request_queue *q, request_fn_proc *rfn,
+			 spinlock_t *lock)
+{
+	return blk_init_allocated_queue_node(q, rfn, lock, -1);
+}
+EXPORT_SYMBOL(blk_init_allocated_queue);
+
+struct request_queue *
+blk_init_allocated_queue_node(struct request_queue *q, request_fn_proc *rfn,
+			      spinlock_t *lock, int node_id)
+{
 	if (!q)
 		return NULL;
 
@@ -601,7 +617,7 @@ blk_init_queue_node(request_fn_proc *rfn, spinlock_t *lock, int node_id)
 	blk_put_queue(q);
 	return NULL;
 }
-EXPORT_SYMBOL(blk_init_queue_node);
+EXPORT_SYMBOL(blk_init_allocated_queue_node);
 
 int blk_get_queue(struct request_queue *q)
 {
block/elevator.c +2 -0
index 76e3702..6ec9137 100644
@@ -921,6 +921,7 @@ int elv_register_queue(struct request_queue *q)
 	}
 	return error;
 }
+EXPORT_SYMBOL_GPL(elv_register_queue);
 
 static void __elv_unregister_queue(struct elevator_queue *e)
 {
@@ -933,6 +934,7 @@ void elv_unregister_queue(struct request_queue *q)
 	if (q)
 		__elv_unregister_queue(q->elevator);
 }
+EXPORT_SYMBOL_GPL(elv_unregister_queue);
 
 void elv_register(struct elevator_type *e)
 {
include/linux/blkdev.h +5 -0
index 6690e8b..89ce7a2 100644
@@ -917,7 +917,12 @@ extern void blk_abort_queue(struct request_queue *);
  */
 extern struct request_queue *blk_init_queue_node(request_fn_proc *rfn,
 					spinlock_t *lock, int node_id);
+extern struct request_queue *blk_init_allocated_queue_node(struct request_queue *,
+							   request_fn_proc *,
+							   spinlock_t *, int node_id);
 extern struct request_queue *blk_init_queue(request_fn_proc *, spinlock_t *);
+extern struct request_queue *blk_init_allocated_queue(struct request_queue *,
+						      request_fn_proc *, spinlock_t *);
 extern void blk_cleanup_queue(struct request_queue *);
 extern void blk_queue_make_request(struct request_queue *, make_request_fn *);
 extern void blk_queue_bounce_limit(struct request_queue *, u64);
Signature · 1 line
1.6.6.1
#1 lore raw

RFC PATCH 2/2 dm: only initialize full request_queue for request-based device

Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.

dm_init_request_based_queue is now called at the end of a request-based DM device's table load.

Otherwise bio-based DM devices will have an elevator, request_fn, etc. As a result bio-based DM devices had an 'iosched/' tree as well as an 'elevator' other than "none" in sysfs.

---
 drivers/md/dm-table.c |    5 ++++
 drivers/md/dm.c       |   52 ++++++++++++++++++++++++++++++++-----------------
 drivers/md/dm.h       |    2 +
 3 files changed, 41 insertions(+), 18 deletions(-)
drivers/md/dm-table.c +5 -0
index 9924ea2..64a8578 100644
@@ -829,6 +829,11 @@ int dm_table_set_type(struct dm_table *t)
 		return -EINVAL;
 	}
 
+	if (!dm_init_request_based_queue(t->md)) {
+		DMWARN("Cannot initialize queue for Request-based dm");
+		return -EINVAL;
+	}
+
 	t->type = DM_TYPE_REQUEST_BASED;
 
 	return 0;
drivers/md/dm.c +34 -18

73 lines not on this page · raw message

drivers/md/dm.h +2 -0
index bad1724..489feba 100644
@@ -113,6 +113,8 @@ void dm_sysfs_exit(struct mapped_device *md);
 struct kobject *dm_kobject(struct mapped_device *md);
 struct mapped_device *dm_get_from_kobject(struct kobject *kobj);
 
+int dm_init_request_based_queue(struct mapped_device *md);
+
 /*
  * Targets for linear and striped mappings
  */
Signature · 1 line
1.6.6.1
#2 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 4 lines · Mike Snitzer

Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.

Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)

As I mentioned on the last email of the thread above (see below), specifying device type at the device creation time by userspace tools should make dm code very simple. So that may be a better approach.

Quoted 11 lines

By the way, another approach to optimizing the memory usage would be to determine whether the dm device is bio-based or request-based at the device creation time, instead of the table binding time. We want the delayed allocation, since kernel can't decide the device type until the first table is bound because of the auto-detection mechanism. The auto-detection is good for keeping compatibility with existing user-space tools. But once user-space tools are changed to specify device type at the device creation time, we can eventually remove the auto-detection.
Then, kernel can decide device type in alloc_dev(), so
the initialization code in kernel will become very simple.

Thanks,
Kiyoshi Ueda

#3 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 8 lines · Kiyoshi Ueda

Hi Mike,

Quoted 4 lines · Mike Snitzer

Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.

Quoted 5 lines

Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)

Thanks for the reference. I completely forgot about that thread (even though I responded to Nikanth's patches in detail! :)

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Interestingly, having to revisit this issue (forgetting that this line of work was already explored) I came up with roughly the same type of change to the block layer as Nikanth's 1/2 patch. The difference being my blk_init_allocated_queue is more minimalist because the block code has evolved to allow this change to be more natural.

Similarly, my proposed DM changes are also quite natural. By using dm_table_set_type() as the hook to initialize the request-based DM device's elevator we perform allocations during table load.

Having just looked at Nikanth's proposed DM patch 2/2 again it shows that blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.

Quoted 18 lines

As I mentioned on the last email of the thread above (see below), specifying device type at the device creation time by userspace tools should make dm code very simple. So that may be a better approach.

Quoted 11 lines

By the way, another approach to optimizing the memory usage would be to determine whether the dm device is bio-based or request-based at the device creation time, instead of the table binding time. We want the delayed allocation, since kernel can't decide the device type until the first table is bound because of the auto-detection mechanism. The auto-detection is good for keeping compatibility with existing user-space tools. But once user-space tools are changed to specify device type at the device creation time, we can eventually remove the auto-detection.
Then, kernel can decide device type in alloc_dev(), so
the initialization code in kernel will become very simple.

Thanks,
Kiyoshi Ueda

#4 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 20 lines · Mike Snitzer
Quoted 8 lines · Kiyoshi Ueda

Hi Mike,

Quoted 4 lines · Mike Snitzer

Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.

Quoted 5 lines

Thank you for working on this.
However, I still disagree with this patch as we discussed on this thread:
http://marc.info/?t=124990138700003&r=1&w=2
(Exporting a part of queue's features may cause some maintenance costs
in future.)

Thanks for the reference. I completely forgot about that thread (even though I responded to Nikanth's patches in detail! :)

Quoted 7 lines

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

By the way, your current patch has a problem below. It needs to be fixed at least.

Quoted 3 lines

Similarly, my proposed DM changes are also quite natural. By using dm_table_set_type() as the hook to initialize the request-based DM device's elevator we perform allocations during table load.

Your patch initializes queue everytime request-based table is loaded. I think that could cause a problem (although I haven't tested your patch yet).

Also, as you know, the table load can be canceled.
So initializing queue at table loading time may cause some weird behaviors for users. For example,

    # dmsetup create --notable bio-based
    # echo "0 100 multipath ..." | dmsetup load bio-based
    # echo "0 100 linear ..."    | dmsetup load bio-based
    # dmsetup resume bio-based
    # ls /sys/block/dm-0/queue/
    ... iosched ...
If you (and Alasdair) think this behavior is acceptable, it might
be OK.  I just feel it's weird though...
Quoted 4 lines

Having just looked at Nikanth's proposed DM patch 2/2 again it shows that blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.

Right, in general.
However, in this special case, I think initializing queue (allocating memory) during resume should be OK as I mentioned like below in:
http://marc.info/?l=linux-kernel&m=124999806420663&w=2

> Generally, dm must not allocate memory during resume because > it may cause a deadlock in no memory situation.
> However, there is no I/O on this device at this point, > so the allocation should be ok for this special case. > I think some comments are needed here to describe that.

So you should be able to take this approach.

Thanks,
Kiyoshi Ueda

#5 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 11 lines · Kiyoshi Ueda

Hi Mike,

Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

Quoted 2 lines

By the way, your current patch has a problem below. It needs to be fixed at least.

Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).

Quoted 7 lines
Quoted 3 lines

Similarly, my proposed DM changes are also quite natural. By using dm_table_set_type() as the hook to initialize the request-based DM device's elevator we perform allocations during table load.

Your patch initializes queue everytime request-based table is loaded. I think that could cause a problem (although I haven't tested your patch yet).

Yes, that was definitely a problem. I've included an incremental patch that shows the fixes to dm_init_request_based_queue below. I'll send out a revised patch: [PATCH 2/2 v2] ...

If you're comfortable with it please respond with your Acked-by.

Quoted 11 lines

Also, as you know, the table load can be canceled.
So initializing queue at table loading time may cause some weird behaviors for users. For example,

    # dmsetup create --notable bio-based
    # echo "0 100 multipath ..." | dmsetup load bio-based
    # echo "0 100 linear ..."    | dmsetup load bio-based
    # dmsetup resume bio-based
    # ls /sys/block/dm-0/queue/
    ... iosched ...
If you (and Alasdair) think this behavior is acceptable, it might
be OK.  I just feel it's weird though...

Agreed, we don't think this is ideal.. I have a follow-on patch that I'll mail out separately for your consideration ([RFC PATCH 3/2]).

Quoted 10 lines
Quoted 4 lines

Having just looked at Nikanth's proposed DM patch 2/2 again it shows that blk_init_allocated_queue(), which allocates memory, was being called during resume (dm_swap_table). Allocations are not allowed during resume.

Right, in general.
However, in this special case, I think initializing queue (allocating memory) during resume should be OK as I mentioned like below in:
http://marc.info/?l=linux-kernel&m=124999806420663&w=2

Quoted 7 lines

> Generally, dm must not allocate memory during resume because > it may cause a deadlock in no memory situation.
> However, there is no I/O on this device at this point, > so the allocation should be ok for this special case. > I think some comments are needed here to describe that.

So you should be able to take this approach.

I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.

Here is the incremental patch I mentioned above.
---

Author: Mike Snitzer <[email protected]>
Date:   Wed May 12 18:52:01 2010 -0400

idempotent dm_init_request_based_queue

drivers/md/dm.c +19 -7
index 6df7f6c..b2171be 100644
@@ -1951,14 +1951,30 @@ bad_module_get:
 	return NULL;
 }
 
+/*
+ * Fully initialize the request_queue (elevator, ->request_fn, etc).
+ */
 int dm_init_request_based_queue(struct mapped_device *md)
 {
 	struct request_queue *q = NULL;
 
-	q = blk_init_allocated_queue(md->queue, dm_request_fn, NULL);
-	if (!q)
+	if (!md->queue)
 		return 0;
-	md->queue = q;
+
+	/*
+	 * Avoid re-initializing the queue (and leaking the existing
+	 * elevator) if dm_init_request_based_queue() was already used.
+	 */
+	if (!md->queue->elevator) {
+		/* Fully initialize the queue */
+		q = blk_init_allocated_queue(md->queue, dm_request_fn, NULL);
+		if (!q)
+			return 0;
+		md->queue = q;
+		md->saved_make_request_fn = md->queue->make_request_fn;
+		blk_queue_make_request(md->queue, dm_request);
+		elv_register_queue(md->queue);
+	}
 
 	/*
 	 * Request-based dm devices cannot be stacked on top of bio-based dm
@@ -1970,7 +1986,6 @@ int dm_init_request_based_queue(struct mapped_device *md)
 	 * This queue is new, so no concurrency on the queue_flags.
 	 */
 	queue_flag_clear_unlocked(QUEUE_FLAG_STACKABLE, md->queue);
-	md->saved_make_request_fn = md->queue->make_request_fn;
 
 	blk_queue_softirq_done(md->queue, dm_softirq_done);
 	blk_queue_prep_rq(md->queue, dm_prep_fn);
@@ -1978,9 +1993,6 @@ int dm_init_request_based_queue(struct mapped_device *md)
 	blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN_FLUSH,
 			  dm_rq_prepare_flush);
 
-	/* Register the request-based queue's elevator with sysfs */
-	elv_register_queue(md->queue);
-
 	return 1;
 }
 
#6 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 23 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

Hi Mike,

Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

Quoted 5 lines
Quoted 2 lines

By the way, your current patch has a problem below. It needs to be fixed at least.

Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).

I found another bug; blk_init_allocated_queue() overwrites q->unplug_fn with generic one. (Although it's currently harmless for multipath target.)

I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.
I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.

Quoted 3 lines

I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.

OK.
Then, how about the patch below?
It still fully initializes queue at device creation time for both bio-based and request-based. Then, it drops elevator when the device type is decided as bio-based at table binding time.
So no memory allocation during resume (freeing instead).

I hope this simple work-around approach is acceptable for you and Alasdair (and others).
(Then, let's focus on making a right fix rather than hacking
the block layer.)

[PATCH] dm: drop elevator when the device is bio-based

I/O scheduler affects nothing for bio-based dm device. Showing I/O scheduler's attributes in sysfs for bio-based dm devices is confusing users.
So drop them from sysfs when a dm device is decided as bio-based.

This patch depends on the commit below in the for-2.6.35 of Jens' linux-2.6-block git:
commit 01effb0dc1451fad55925873ffbfb88fa4eadce0

Author: Mike Snitzer <[email protected]>
Date:   Tue May 11 08:57:42 2010 +0200

block: allow initialization of previously allocated request_queue

---
 drivers/md/dm.c |    8 ++++++++
 1 file changed, 8 insertions(+)
drivers/md/dm.c +8 -0
Index: 2.6.34-rc7/drivers/md/dm.c
===================================================================
@@ -2410,6 +2410,14 @@ struct dm_table *dm_swap_table(struct ma
 		goto out;
 	}
 
+	/* drop elevator when the device type is decided as bio-based */
+	if (!md->map && dm_table_get_type(table) == DM_TYPE_BIO_BASED) {
+		elv_unregister_queue(md->queue);
+		elevator_exit(md->queue->elevator);
+		md->queue->request_fn = NULL;
+		md->queue->elevator = NULL;
+	}
+
 	map = __bind(md, table, &limits);
 
 out:
#7 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 26 lines · Kiyoshi Ueda

Hi Mike,

Quoted 23 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

Hi Mike,

Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

By the way, your current patch has a problem below. It needs to be fixed at least.

Thanks for the review. I've addressed both your points with additional changes (split between 2 patches).

I found another bug; blk_init_allocated_queue() overwrites q->unplug_fn with generic one. (Although it's currently harmless for multipath target.)

Thanks again for your review!

Here is a simple incremental fix:

drivers/md/dm.c +13 -8
index 5390754..985dd9c 100644
@@ -1854,6 +1854,17 @@ static const struct block_device_operations dm_blk_dops;
 static void dm_wq_work(struct work_struct *work);
 static void dm_rq_barrier_work(struct work_struct *work);
 
+static void dm_init_md_queue(struct mapped_device *md)
+{
+	md->queue->queuedata = md;
+	md->queue->backing_dev_info.congested_fn = dm_any_congested;
+	md->queue->backing_dev_info.congested_data = md;
+	blk_queue_make_request(md->queue, dm_request);
+	blk_queue_bounce_limit(md->queue, BLK_BOUNCE_ANY);
+	md->queue->unplug_fn = dm_unplug_all;
+	blk_queue_merge_bvec(md->queue, dm_merge_bvec);
+}
+
 /*
  * Allocate and initialise a blank device with a given minor.
  */
@@ -1895,13 +1906,7 @@ static struct mapped_device *alloc_dev(int minor)
 	if (!md->queue)
 		goto bad_queue;
 
-	md->queue->queuedata = md;
-	md->queue->backing_dev_info.congested_fn = dm_any_congested;
-	md->queue->backing_dev_info.congested_data = md;
-	blk_queue_make_request(md->queue, dm_request);
-	blk_queue_bounce_limit(md->queue, BLK_BOUNCE_ANY);
-	md->queue->unplug_fn = dm_unplug_all;
-	blk_queue_merge_bvec(md->queue, dm_merge_bvec);
+	dm_init_md_queue(md);
 
 	md->disk = alloc_disk(1);
 	if (!md->disk)
@@ -1974,7 +1979,7 @@ int dm_init_request_based_queue(struct mapped_device *md)
 			return 0;
 		md->queue = q;
 		md->saved_make_request_fn = md->queue->make_request_fn;
-		blk_queue_make_request(md->queue, dm_request);
+		dm_init_md_queue(md);
 		register_elevator = 1;
 	} else if (dm_bio_based_md(md)) {
 		/*
Quoted 2 lines

I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.

Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.

The longer-term direction you want to take DM (known type during alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize the request_queue for a DM device (whether it is bio-based or request-based).

My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.

I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.

Quoted 8 lines

I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.

Quoted 3 lines

I confirmed with Alasdair that we don't want to conditionally relax DM's allocation constraints for request-based DM. Best to be consistent about not allowing allocations during resume.

Quoted 6 lines

OK.
Then, how about the patch below?
It still fully initializes queue at device creation time for both bio-based and request-based. Then, it drops elevator when the device type is decided as bio-based at table binding time.
So no memory allocation during resume (freeing instead).

I'm inclined to believe that the extra work for bio-based DM is not ideal. And even freeing during resume _could_ pose a problem during a resume (if it triggers unknown additional housekeeping). elv_unregister_queue() and elevator_exit() do quite a bit more than simply freeing memory. We can't safely assume those methods will never allocate memory to accomplish their cleanup (now or in the future).

Quoted 2 lines

I hope this simple work-around approach is acceptable for you and Alasdair (and others).

While it is certainly a small change it carries with it the disadvantage of still requiring more work than is needed for bio-based request_queue initialization. It also poses a risk by requiring additional work when resuming a bio-based device.

Quoted 2 lines

(Then, let's focus on making a right fix rather than hacking
the block layer.)

I haven't been hacking the block layer. Please don't misrepresent what I proposed.

Regards,
Mike

#8 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 29 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda

Hi Mike,

Quoted 23 lines · Mike Snitzer
Quoted 11 lines · Kiyoshi Ueda

Hi Mike,

Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

snip

Quoted 15 lines
Quoted 6 lines

I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.

I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.

Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.

The longer-term direction you want to take DM (known type during alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize the request_queue for a DM device (whether it is bio-based or request-based).

Quoted 4 lines

My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.

As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.

Your patch makes that loading table affects the block device, so you are changing the model.

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.

My suggestion was (and is) to change the current model by determining bio-based/request-based at device creation time, not by a table.

Quoted 3 lines

I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.

The number of patches doesn't matter.
I still feel it's becoming complex, for example it now has to take care of reverting md/queue changes by previous preload.

Also, you have to take care of race condition between concurrent table loadings which was mentioned above, because the table which was used to initialized the queue may not be set in hc->new_map. Although such races may not happen in real usages, they could cause some critical problems (maybe kernel panic, maybe memory leak).

Thanks,
Kiyoshi Ueda

#9 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 60 lines · Kiyoshi Ueda

Hi Mike,

Quoted 29 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda

Hi Mike,

Quoted 23 lines · Mike Snitzer
Quoted 11 lines · Kiyoshi Ueda

Hi Mike,

Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

snip

Quoted 15 lines
Quoted 6 lines

I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.

I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.

Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.

The longer-term direction you want to take DM (known type during alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize the request_queue for a DM device (whether it is bio-based or request-based).

Quoted 4 lines

My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.

As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.

Quoted 8 lines

Your patch makes that loading table affects the block device, so you are changing the model.

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.

Nice catch, yes md->queue needs protection (see patch below).

Quoted 11 lines

My suggestion was (and is) to change the current model by determining bio-based/request-based at device creation time, not by a table.

Quoted 3 lines

I'll post a single patch that contains my changes (no need to have 2 patches any more). With that patch I'm hopeful you'll see my changes aren't as complex as they might have seemed over the past few days.

The number of patches doesn't matter.
I still feel it's becoming complex, for example it now has to take care of reverting md/queue changes by previous preload.

AFAIK, no md/queue changes would need to be reverted. The last table load wins and the DM device's queue is left in the correct state for that winning table (assuming the patch at the end of this mail).

As for the overall complexity of my proposed changes, I'm inclined to address this issue without userspace changes. But I do understand we're at a cross-roads (and likely need Alasdair, and/or others, to weigh in):

1) Are my proposed kernel-only changes somehow detrimental to DM right now?
- We cannot let known bugs go unfixed because of unknown future requirements.

2) Is requiring the DM create ioctl to know that a device will be request-based vs bio-based really a reasonable requirement?
- To me a DM device's queue type is a DM (kernel) implementation detail that userspace shouldn't _ever_ need to know about. If userspace knew it'd make the kernel's life easier but that doesn't make it the correct solution on its own.

Quoted 5 lines

Also, you have to take care of race condition between concurrent table loadings which was mentioned above, because the table which was used to initialized the queue may not be set in hc->new_map. Although such races may not happen in real usages, they could cause some critical problems (maybe kernel panic, maybe memory leak).

In the worst case, the hypothetical concurrent table loadings (of conflicting types: request vs bio) will just cause the request_queue to be fully initialized (allocated elevator) for bio-based devices. There aren't any other concerns with panics or memory leaks. Granted something like the following patch is required:

drivers/md/dm.c +20 -4
index 143082c..02966ef 100644
@@ -123,6 +123,11 @@ struct mapped_device {
 	unsigned long flags;
 
 	struct request_queue *queue;
+	/*
+	 * Protect queue from concurrent initialization during
+	 * table load
+	 */
+	struct mutex queue_lock;
 	struct gendisk *disk;
 	char name[16];
 
@@ -1892,6 +1897,7 @@ static struct mapped_device *alloc_dev(int minor)
 
 	init_rwsem(&md->io_lock);
 	mutex_init(&md->suspend_lock);
+	mutex_init(&md->queue_lock);
 	spin_lock_init(&md->deferred_lock);
 	spin_lock_init(&md->barrier_error_lock);
 	rwlock_init(&md->map_lock);
@@ -1966,14 +1972,16 @@ bad_module_get:
  */
 int dm_init_request_based_queue(struct mapped_device *md)
 {
+	int r = 0;
 	struct request_queue *q = NULL;
 
+	mutex_lock(&md->queue_lock);
 	/* Avoid re-initializing the queue if already fully initialized */
 	if (!md->queue->elevator) {
 		/* Fully initialize the queue */
 		q = blk_init_allocated_queue(md->queue, dm_request_fn, NULL);
 		if (!q)
-			return 0;
+			goto out;
 		md->queue = q;
 		md->saved_make_request_fn = md->queue->make_request_fn;
 		dm_init_md_queue(md);
@@ -1984,8 +1992,11 @@ int dm_init_request_based_queue(struct mapped_device *md)
 		 * bio-based back to request-based, e.g.: rq -> bio -> rq
 		 */
 		md->queue->request_fn = dm_request_fn;
-	} else
-		return 1; /* already request-based */
+	} else {
+		/* already request-based */
+		r = 1;
+		goto out;
+	}
 
 	elv_register_queue(md->queue);
 
@@ -2006,7 +2017,10 @@ int dm_init_request_based_queue(struct mapped_device *md)
 	blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN_FLUSH,
 			  dm_rq_prepare_flush);
 
-	return 1;
+	r = 1;
+out:
+	mutex_unlock(&md->queue_lock);
+	return r;
 }
 
 void dm_clear_request_based_queue(struct mapped_device *md)
@@ -2015,8 +2029,10 @@ void dm_clear_request_based_queue(struct mapped_device *md)
 		return; /* already bio-based */
 
 	/* Unregister elevator from sysfs and clear ->request_fn */
+	mutex_lock(&md->queue_lock);
 	elv_unregister_queue(md->queue);
 	md->queue->request_fn = NULL;
+	mutex_unlock(&md->queue_lock);
 }
 
 static void unlock_fs(struct mapped_device *md);
#10 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 54 lines · Mike Snitzer
Quoted 53 lines · Kiyoshi Ueda
Quoted 21 lines · Mike Snitzer
Quoted 20 lines · Kiyoshi Ueda
Quoted 19 lines · Mike Snitzer
Quoted 9 lines · Kiyoshi Ueda
Quoted 7 lines · Mike Snitzer

It is clear we need to resolve the current full request_queue initialization that occurs even for bio-based DM devices.

I believe the 2 patches I posted accomplish this in a stright-forward way. We can always improve on it (by looking at what you proposed below) but we need a minimlaist fix that doesn't depend on userspace LVM2 changes right now.

Quoted 5 lines

Humm, OK.
Indeed, showing iosched directory in bio-based device's sysfs is confusing users actually, and we need something to resolve that soon. So I don't strongly object to your approach as the first step, as long as we can accept the risk of the maintenance cost which I mentioned.

OK, I understand your concern (especially after having gone through this further over the past couple days). I'm hopeful maintenance will be minimal.

snip

Quoted 15 lines
Quoted 6 lines

I feel your patch-set is growing and becoming complex fix rather than minimalist/simple fix.

I think touching mapped_device/queue at table loading time makes things complex. It is natural that table's arguments are reflected to mapped_device/queue at table binding time instead.

Seems the numerous patches I've posted over the past couple days have given a false impression. But I do understand your concern.

The longer-term direction you want to take DM (known type during alloc_dev) illustrates that we share a common goal: only do the precise work that is needed to initialize the request_queue for a DM device (whether it is bio-based or request-based).

Quoted 4 lines

My changes do accomplish that in the near-term without requiring userspace change. Many of my proposed changes are just refactoring existing code to clearly split out what is needed for request-based vs bio-based.

As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.

Your patch makes that loading table affects the block device, so you are changing the model.

Quoted 7 lines
Quoted 5 lines

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.

Nice catch, yes md->queue needs protection (see patch below).

Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of the hash_cell for the mapped_device in protection by _hash_lock.

Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the mapped_device while the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.

Thanks,
Kiyoshi Ueda

#11 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 28 lines · Kiyoshi Ueda

Hi Mike,

Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.

Your patch makes that loading table affects the block device, so you are changing the model.

Quoted 7 lines
Quoted 5 lines

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.

Nice catch, yes md->queue needs protection (see patch below).

Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of the hash_cell for the mapped_device in protection by _hash_lock.

Quoted 4 lines

Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the mapped_device while the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.

Ah, yes.. I looked at the possibility of serializing the entirety of table_load but determined that would be too excessive (would reduce parallelism of table_load). But I clearly missed the fact that there could be a race to the _hash_lock protected critical section in table_load() -- leading to queue inconsistency.

I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load(). So it sets the queue up _after_ the table's type is established with dm_table_set_type().

Thanks again for your review; I'll work to be more meticulous in the future!

drivers/md/dm-ioctl.c +8 -0
index d7500e1..bfb283c 100644
@@ -1180,6 +1180,14 @@ static int table_load(struct dm_ioctl *param, size_t param_size)
 		goto out;
 	}
 
+	r = dm_table_setup_md_queue(t);
+	if (r) {
+		DMWARN("unable to setup device queue for this table");
+		dm_table_destroy(t);
+		up_write(&_hash_lock);
+		goto out;
+	}
+
 	if (hc->new_map)
 		dm_table_destroy(hc->new_map);
 	hc->new_map = t;
drivers/md/dm-table.c +17 -6
index dec8295..990cf17 100644
@@ -803,7 +803,6 @@ int dm_table_set_type(struct dm_table *t)
 	if (bio_based) {
 		/* We must use this table as bio-based */
 		t->type = DM_TYPE_BIO_BASED;
-		dm_clear_request_based_queue(t->md);
 		return 0;
 	}
 
@@ -830,11 +829,6 @@ int dm_table_set_type(struct dm_table *t)
 		return -EINVAL;
 	}
 
-	if (!dm_init_request_based_queue(t->md)) {
-		DMWARN("Cannot initialize queue for Request-based dm");
-		return -EINVAL;
-	}
-
 	t->type = DM_TYPE_REQUEST_BASED;
 
 	return 0;
@@ -850,6 +844,23 @@ bool dm_table_request_based(struct dm_table *t)
 	return dm_table_get_type(t) == DM_TYPE_REQUEST_BASED;
 }
 
+/*
+ * Setup the DM device's queue based on table's type.
+ * Caller must serialize access to table's md.
+ */
+int dm_table_setup_md_queue(struct dm_table *t)
+{
+	if (dm_table_request_based(t)) {
+		if (!dm_init_request_based_queue(t->md)) {
+			DMWARN("Cannot initialize queue for Request-based dm");
+			return -EINVAL;
+		}
+	} else
+		dm_clear_request_based_queue(t->md);
+
+	return 0;
+}
+
 int dm_table_alloc_md_mempools(struct dm_table *t)
 {
 	unsigned type = dm_table_get_type(t);
drivers/md/dm.c +11 -11
index 143082c..76b61ca 100644
@@ -1856,6 +1856,17 @@ static void dm_rq_barrier_work(struct work_struct *work);
 
 static void dm_init_md_queue(struct mapped_device *md)
 {
+	/*
+	 * Request-based dm devices cannot be stacked on top of bio-based dm
+	 * devices.  The type of this dm device has not been decided yet.
+	 * The type is decided at the first table loading time.
+	 * To prevent problematic device stacking, clear the queue flag
+	 * for request stacking support until then.
+	 *
+	 * This queue is new, so no concurrency on the queue_flags.
+	 */
+	queue_flag_clear_unlocked(QUEUE_FLAG_STACKABLE, md->queue);
+
 	md->queue->queuedata = md;
 	md->queue->backing_dev_info.congested_fn = dm_any_congested;
 	md->queue->backing_dev_info.congested_data = md;
@@ -1989,17 +2000,6 @@ int dm_init_request_based_queue(struct mapped_device *md)
 
 	elv_register_queue(md->queue);
 
-	/*
-	 * Request-based dm devices cannot be stacked on top of bio-based dm
-	 * devices.  The type of this dm device has not been decided yet.
-	 * The type is decided at the first table loading time.
-	 * To prevent problematic device stacking, clear the queue flag
-	 * for request stacking support until then.
-	 *
-	 * This queue is new, so no concurrency on the queue_flags.
-	 */
-	queue_flag_clear_unlocked(QUEUE_FLAG_STACKABLE, md->queue);
-
 	blk_queue_softirq_done(md->queue, dm_softirq_done);
 	blk_queue_prep_rq(md->queue, dm_prep_fn);
 	blk_queue_lld_busy(md->queue, dm_lld_busy);
drivers/md/dm.h +1 -0
index 95aec64..b4ebb11 100644
@@ -65,6 +65,7 @@ bool dm_table_request_based(struct dm_table *t);
 int dm_table_alloc_md_mempools(struct dm_table *t);
 void dm_table_free_md_mempools(struct dm_table *t);
 struct dm_md_mempools *dm_table_get_md_mempools(struct dm_table *t);
+int dm_table_setup_md_queue(struct dm_table *t);
 
 /*
  * To check the return value from dm_table_find_target().
#12 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 38 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

As far as I understand, the current model of device-mapper is:
- a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
- a table and its attributes are bound to the block device on resume
If we want to fix a problem, I think we should either work based on this model or change the model.

Your patch makes that loading table affects the block device, so you are changing the model.

Quoted 7 lines
Quoted 5 lines

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself. If you change this model, table loading needs to be serialized with appropriate locking.

Nice catch, yes md->queue needs protection (see patch below).

Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
1. populate table
2. set the table to ->new_map of the hash_cell for the mapped_device in protection by _hash_lock.

Quoted 4 lines

Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the mapped_device while the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.

Ah, yes.. I looked at the possibility of serializing the entirety of table_load but determined that would be too excessive (would reduce parallelism of table_load). But I clearly missed the fact that there could be a race to the _hash_lock protected critical section in table_load() -- leading to queue inconsistency.

Quoted 6 lines

I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load(). So it sets the queue up _after_ the table's type is established with dm_table_set_type().

dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Thanks,
Kiyoshi Ueda

#13 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 48 lines · Kiyoshi Ueda

Hi Mike,

Quoted 38 lines · Mike Snitzer
Quoted 26 lines · Kiyoshi Ueda
Quoted 11 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

As far as I understand, the current model of device-mapper is:
  - a table (precisely, a target) has various attributes, bio-based/request-based is one of such attributes
  - a table and its attributes are bound to the block device on resume If we want to fix a problem, I think we should either work based on this model or change the model.

Your patch makes that loading table affects the block device, so you are changing the model.

Quoted 7 lines
Quoted 5 lines

If you change the model, it should be done carefully.
For example, the current model allows most of the table loading code to run without exclusive lock on the device because it doesn't affect the device itself.  If you change this model, table loading needs to be serialized with appropriate locking.

Nice catch, yes md->queue needs protection (see patch below).

Not enough. (See drivers/md/dm-ioctl.c:table_load().)
Table load sequence is:
  1. populate table
  2. set the table to ->new_map of the hash_cell for the mapped_device in protection by _hash_lock.

Quoted 4 lines

Since your fix only serializes the step 1, concurrent table loading could end up with inconsistent status; e.g. request-based table is bound to the mapped_device while the queue is initialized as bio-based. With your new model, those 2 steps above must be atomic.

Ah, yes.. I looked at the possibility of serializing the entirety of table_load but determined that would be too excessive (would reduce parallelism of table_load).  But I clearly missed the fact that there could be a race to the _hash_lock protected critical section in table_load() -- leading to queue inconsistency.

Quoted 6 lines

I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4.  v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load().  So it sets the queue up _after_ the table's type is established with dm_table_set_type().

Quoted 4 lines

dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.

We make no guarantees that other DM devices will resume before a table load -- so calling dm_table_setup_md_queue() within the exclusive _hash_lock is no different than other DM devices being suspended while a request-based DM device performs its first table_load().

My thinking was this should not be a problem as it is only valid to call dm_table_setup_md_queue() before the newly created request-based DM device has been resumed.

AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any GFP_KERNEL allocation could recurse (elevator_alloc() currently uses GFP_KERNEL with kmalloc_node)...

I'll have to review the DM code further to see if all memory allocations during table_load() are done via mempools. I'll also bring this up on this week's LVM call.

Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used.  (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

Thanks,
Mike

#14 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 25 lines · Mike Snitzer
Quoted 10 lines · Kiyoshi Ueda

Hi Mike,

Quoted 6 lines · Mike Snitzer

I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4.  v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load().  So it sets the queue up _after_ the table's type is established with dm_table_set_type().

Quoted 4 lines

dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.

We make no guarantees that other DM devices will resume before a table load -- so calling dm_table_setup_md_queue() within the exclusive _hash_lock is no different than other DM devices being suspended while a request-based DM device performs its first table_load().

My thinking was this should not be a problem as it is only valid to call dm_table_setup_md_queue() before the newly created request-based DM device has been resumed.

Quoted 8 lines

AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any GFP_KERNEL allocation could recurse (elevator_alloc() currently uses GFP_KERNEL with kmalloc_node)...

I'll have to review the DM code further to see if all memory allocations during table_load() are done via mempools. I'll also bring this up on this week's LVM call.

We discussed this and I understand the scope of the problem now.

Just reiterating what you covered when you first pointed this issue out:

It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive _hash_lock while blocking. Having _hash_lock prevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

I hope to have v6 available at some point today but it may be delayed by a day.

Quoted 5 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used.  (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Quoted 8 lines

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

Regards,
Mike

#15 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 7 lines · Mike Snitzer
Quoted 7 lines · Kiyoshi Ueda
Quoted 6 lines

Hi Mike,

dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.

<snip>

Quoted 11 lines

We discussed this and I understand the scope of the problem now.

Just reiterating what you covered when you first pointed this issue out:

It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive _hash_lock while blocking. Having _hash_lock prevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.

Quoted 3 lines

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But

s/multiple_device/mapped_device/

Mike

#16 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

In reply to Mike Snitzer

Hi Mike,

Quoted 35 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Kiyoshi Ueda
Quoted 6 lines · Mike Snitzer

I'll post v5 of the overall patch which will revert the mapped_device 'queue_lock' serialization that I proposed in v4. v5 will contain the following patch to localize all table load related queue manipulation to the _hash_lock protected critical section in table_load(). So it sets the queue up _after_ the table's type is established with dm_table_set_type().

Quoted 4 lines

dm_table_setup_md_queue() may allocate memory with blocking mode. Blocking allocation inside exclusive _hash_lock can cause deadlock; e.g. when it has to wait for other dm devices to resume to free some memory.

We make no guarantees that other DM devices will resume before a table load -- so calling dm_table_setup_md_queue() within the exclusive _hash_lock is no different than other DM devices being suspended while a request-based DM device performs its first table_load().

My thinking was this should not be a problem as it is only valid to call dm_table_setup_md_queue() before the newly created request-based DM device has been resumed.

Quoted 8 lines

AFAIK we don't have any explicit constraints on memory allocations during table load (e.g. table loads shouldn't depend on other devices' writeback) -- but any GFP_KERNEL allocation could recurse (elevator_alloc() currently uses GFP_KERNEL with kmalloc_node)...

I'll have to review the DM code further to see if all memory allocations during table_load() are done via mempools. I'll also bring this up on this week's LVM call.

We discussed this and I understand the scope of the problem now.

Quoted 8 lines

Just reiterating what you covered when you first pointed this issue out:

It could be that a table load gets blocked (waiting on a memory allocation). The table load can take as long as it needs. But we can't have it block holding the exclusive _hash_lock while blocking. Having _hash_lock prevents further DM ioctls. The table load's allocation may be blocking waiting for writeback to a DM device that will be resumed by another thread.

That's right.

Quoted 4 lines

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

You have to see do_resume(), too.
do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL
                                       down_write(_hash_lock)
                                       new_map = hc->new_map (<= rq-based)
                                       up_write(_hash_lock)
                                       dm_swap_table(md, new_map)
                                       __bind(md, new_map) (<= rq-based)

down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)

Other ioctls might be as well, since I have checked only interaction between table load and resume.

Quoted 2 lines

I hope to have v6 available at some point today but it may be delayed by a day.

Sorry for repeating but since you are trying to change the current dm model, please design/consider it carefully; what objects/ioctls are involved and how you change their relationships, where are the critical sections and how you protect them.
Otherwise, I'm afraid we will just spend your time in transforming one bug to another. (and my time for reviewing and verifying problems :)

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

Quoted 7 lines
Quoted 5 lines

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL

Thanks,
Kiyoshi Ueda

#17 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 22 lines · Kiyoshi Ueda

Hi Mike,

Quoted 4 lines · Mike Snitzer

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

You have to see do_resume(), too.
do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL
                                       down_write(_hash_lock)
                                       new_map = hc->new_map (<= rq-based)
                                       up_write(_hash_lock)
                                       dm_swap_table(md, new_map)
                                       __bind(md, new_map) (<= rq-based)
Quoted 4 lines

down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)

Understood.

Quoted 2 lines

Other ioctls might be as well, since I have checked only interaction between table load and resume.

Table load and resume should be the only ioctls that are relevant.

Quoted 9 lines
Quoted 2 lines

I hope to have v6 available at some point today but it may be delayed by a day.

Sorry for repeating but since you are trying to change the current dm model, please design/consider it carefully; what objects/ioctls are involved and how you change their relationships, where are the critical sections and how you protect them.
Otherwise, I'm afraid we will just spend your time in transforming one bug to another. (and my time for reviewing and verifying problems :)

Noted. I do appreciate your thorough reviews but will hopefully reduce your burden by improving my awareness and code.

Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

Quoted 7 lines
Quoted 5 lines

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)

Quoted 2 lines
    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL

Understood.

I've addressed the above races by:
1) properly protecting md->queue during do_resume(). Both do_resume() and table_load() first take md->queue_lock and then _hash_lock.
2) adding a negative check for an existing live table to dm_table_setup_md_queue().

I will mail out v7 after I've given it additional review.

FYI, I've revised and split out the conflicting table type checking patch as that patch should not be controversial and is completely independent of this DM queue initialization work. I'll email it to dm-devel soon.

Regards,
Mike

#18 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 25 lines · Mike Snitzer
Quoted 24 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

You have to see do_resume(), too.
do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL
                                       down_write(_hash_lock)
                                       new_map = hc->new_map (<= rq-based)
                                       up_write(_hash_lock)
                                       dm_swap_table(md, new_map)
                                       __bind(md, new_map) (<= rq-based)

down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)

snip

Quoted 31 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

Quoted 7 lines
Quoted 5 lines

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)

Quoted 2 lines
    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL

Understood.

Quoted 7 lines

I've addressed the above races by:
1) properly protecting md->queue during do_resume(). Both do_resume() and table_load() first take md->queue_lock and then _hash_lock.
2) adding a negative check for an existing live table to dm_table_setup_md_queue().

I will mail out v7 after I've given it additional review.

dm_table_setup_md_queue() may allocate memory with blocking mode inside md->queue_lock which is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded in hc->new_map and no live table yet in a mapped_device.
- Load a request-based table and then no memory situation happens in dm_table_setup_md_queue().
- Then, the mapped_device can't be resumed even in the case that resuming the mapped_device with the preloaded bio-based table can make some memory by writeback.
=> deadlock

Thanks,
Kiyoshi Ueda

#19 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Quoted 68 lines · Kiyoshi Ueda

Hi Mike,

Quoted 25 lines · Mike Snitzer
Quoted 24 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

You have to see do_resume(), too.
do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL
                                       down_write(_hash_lock)
                                       new_map = hc->new_map (<= rq-based)
                                       up_write(_hash_lock)
                                       dm_swap_table(md, new_map)
                                       __bind(md, new_map) (<= rq-based)

down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)

snip

Quoted 31 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

Quoted 7 lines
Quoted 5 lines

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)

Quoted 2 lines
    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL

Understood.

Quoted 7 lines

I've addressed the above races by:
1) properly protecting md->queue during do_resume(). Both do_resume() and table_load() first take md->queue_lock and then _hash_lock.
2) adding a negative check for an existing live table to dm_table_setup_md_queue().

I will mail out v7 after I've given it additional review.

Quoted 12 lines

dm_table_setup_md_queue() may allocate memory with blocking mode inside md->queue_lock which is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded in hc->new_map and no live table yet in a mapped_device.
- Load a request-based table and then no memory situation happens in dm_table_setup_md_queue().
- Then, the mapped_device can't be resumed even in the case that resuming the mapped_device with the preloaded bio-based table can make some memory by writeback.
=> deadlock

My patch introduces a new mutex that is local to the mapped_device being loaded. And that same mutex is taken during do_resume (just like you implicitly said was needed to avoid races).

But I'm _very_ skeptical of the validity of this deadlock scenario. You're leaving out some important context for why we might expect a DM device to service (queue) IO during the table load _before_ the device has _ever_ been resumed. When I issue IO to a DM device that only has an inactive table it returns immediately:

# dmsetup table --inactive
bio-based: 0 8388608 linear 254:16 384
# dd if=/dev/mapper/bio-based of=/dev/null bs=1024k count=10
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00016418 s, 0.0 kB/s

As we discussed earlier in this thread, a table load can block indefinitely (waiting for memory) as long as _hash_lock is not held (so other DM ioctls are still possible).

It is a serious reach to suggest that your new deadlock scenario is a legitimate concern. This would mean that a user would:

1) mistakenly load a bio-based table when they meant rq-based
2) load rq-based -- but block waiting for elevator_alloc()
3) attempt to resume device with bio-based anyway -- to make forward progress with writeback destined to the device being resumed for the first time? -- but again, why would dirty pages be destined for this inactive device already?
4) even if the user could resume they'd be left with a bio-based device; which isn't what they wanted..

Do you see the obscure nature of the scenario you've presented?

Please help me understand what I am missing?

Mike

#20 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

Hi Mike,

Quoted 75 lines · Mike Snitzer
Quoted 61 lines · Kiyoshi Ueda
Quoted 24 lines · Mike Snitzer
Quoted 23 lines · Kiyoshi Ueda
Quoted 4 lines · Mike Snitzer

Thanks again for pointing this out; I'll work to arrive at an alternative locking scheme. Likely introduce a lock local to the multiple_device (effectively the 'queue_lock' I had before). But difference is I'd take that lock before taking _hash_lock.

You have to see do_resume(), too.
do_resume() gets hc->new_map with _hash_lock held but without your 'queue_lock' held. This could cause inconsistent status;
table_load() for bio-based table do_resume() for rq-based table
--------------------------------------------------------------------
dm_lock_md_queue(md)
dm_table_setup_md_queue(t) (<= bio-based)

    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL
                                       down_write(_hash_lock)
                                       new_map = hc->new_map (<= rq-based)
                                       up_write(_hash_lock)
                                       dm_swap_table(md, new_map)
                                       __bind(md, new_map) (<= rq-based)

down_write(_hash_lock)
hc->new_map = t
up_write(_hash_lock)
dm_unlock_md_queue(md)

snip

Quoted 27 lines
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Also, your patch changes the queue configuration even when a table is already active and used. (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

Quoted 6 lines
Quoted 5 lines

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

I'll look to address this issue in v6 too.

It can race between table_load() and do_resume() as well;
table_load() for bio-based do_resume() for rq-based
---------------------------------------------------------------------
dm_table_set_type(t) (<= bio-based)
live_table = dm_get_live_table(md)
(assume no live_table yet)
new_map = hc->new_map (<= rq-based) dm_swap_table(md, new_map) __bind(md, new_map)
dm_table_setup_md_queue(t)

Quoted 2 lines
    dm_clear_request_based_queue(md)
      md->queue->request_fn = NULL

Understood.

Quoted 7 lines

I've addressed the above races by:
1) properly protecting md->queue during do_resume(). Both do_resume() and table_load() first take md->queue_lock and then _hash_lock.
2) adding a negative check for an existing live table to dm_table_setup_md_queue().

I will mail out v7 after I've given it additional review.

Quoted 12 lines

dm_table_setup_md_queue() may allocate memory with blocking mode inside md->queue_lock which is needed to resume the md. That could cause a deadlock which doesn't happen in the current model; e.g:
- Assume a bio-based table has been already loaded in hc->new_map and no live table yet in a mapped_device.
- Load a request-based table and then no memory situation happens in dm_table_setup_md_queue().
- Then, the mapped_device can't be resumed even in the case that resuming the mapped_device with the preloaded bio-based table can make some memory by writeback.
=> deadlock

Quoted 8 lines

My patch introduces a new mutex that is local to the mapped_device being loaded. And that same mutex is taken during do_resume (just like you implicitly said was needed to avoid races).

But I'm _very_ skeptical of the validity of this deadlock scenario. You're leaving out some important context for why we might expect a DM device to service (queue) IO during the table load _before_ the device has _ever_ been resumed.

Because upper device might wait for the DM device's resume to serve pending I/O.

In a stacked configuration, load/suspend/resume happens in this order:
1. load tables to upper and lower(s)
2. suspend upper
3. suspend lower(s)
4. resume lower(s)
5. resume upper
And "lower(s)" may include a new DM device which has never been resumed before the step 4.

So, the following example could cause the deadlock scenario:

  # echo "0 10 linear /dev/sda 0" | dmsetup create lower0
  # echo "0 10 linear /dev/mapper/lower0 0" | dmsetup create upper
  # dmsetup create --notable lower1
  # echo "0 10 linear /dev/mapper/lower1 0" | dmsetup load upper
  # echo "0 10 linear /dev/sdb 0" dmsetup load lower1
  <Any I/Os to /dev/mapper/upper is possible here>
  # dmsetup suspend upper
  <I/Os are queued on /dev/mapper/upper>
  # echo "0 10 multipath ../dev/sda../dev/sdb.." | dmsetup load lower1

If no memory happens here, the system may be hanged up because /dev/mapper/lower1 can't be resumed where /dev/mapper/lower1 will be needed for writeback for /dev/mapper/upper when /dev/mapper/upper is resumed.
(Without your patch, /dev/mapper/upper and /dev/mapper/lower1 could
be resumed using pre-loaded bio-based tables and the system could get some memory and be alive.)

Quoted 7 lines

As we discussed earlier in this thread, a table load can block indefinitely (waiting for memory) as long as _hash_lock is not held (so other DM ioctls are still possible).

It is a serious reach to suggest that your new deadlock scenario is a legitimate concern. This would mean that a user would:

Quoted 8 lines

1) mistakenly load a bio-based table when they meant rq-based
2) load rq-based -- but block waiting for elevator_alloc()
3) attempt to resume device with bio-based anyway -- to make forward progress with writeback destined to the device being resumed for the first time? -- but again, why would dirty pages be destined for this inactive device already?
4) even if the user could resume they'd be left with a bio-based device; which isn't what they wanted..

Although the scenario/example which I mentioned above may be too intentional, it is still possible.
We should design not to cause such critical problems or prevent/reject such problematic operations.

Your new model, which determines md type at the first table loading time and prevents type switching, is close to my initial idea, which determines md type at device creation time, and no userspace change is required. I think it's reasonable model.
But I have some comments on that, so I'll reply to the new thread of your new model patch-set.

Thanks,
Kiyoshi Ueda

#21 lore raw

Re: [RFC PATCH 2/2] dm: only initialize full request_queue for request-based device

In reply to Mike Snitzer
Quoted 6 lines · Mike Snitzer
Quoted 4 lines · Kiyoshi Ueda

Also, your patch changes the queue configuration even when a table is already active and used.  (e.g. Loading bio-based table to a mapped_device which is already active/used as request-based sets q->requst_fn in NULL.) That could cause some critical problems.

Quoted 8 lines

Yes, that is possible and I can add additional checks to prevent this. But this speaks to a more general problem with the existing DM code.

dm_swap_table() has the negative check to prevent such table loads, e.g.:
/* cannot change the device type, once a table is bound */

This check should come during table_load, as part of dm_table_set_type(), rather than during table resume.

FYI, the following patch is the relevant portion of the v6 patch that addresses the issue discussed above.

drivers/md/dm-table.c +14 -1
index 990cf17..62ebd94 100644
@@ -782,6 +782,7 @@ int dm_table_set_type(struct dm_table *t)
 {
 	unsigned i;
 	unsigned bio_based = 0, request_based = 0;
+	struct dm_table *live_table = NULL;
 	struct dm_target *tgt;
 	struct dm_dev_internal *dd;
 	struct list_head *devices;
@@ -803,7 +804,7 @@ int dm_table_set_type(struct dm_table *t)
 	if (bio_based) {
 		/* We must use this table as bio-based */
 		t->type = DM_TYPE_BIO_BASED;
-		return 0;
+		goto finish;
 	}
 
 	BUG_ON(!request_based); /* No targets in this table */
@@ -831,6 +832,18 @@ int dm_table_set_type(struct dm_table *t)
 
 	t->type = DM_TYPE_REQUEST_BASED;
 
+finish:
+	/* cannot change the device type, once a table is bound */
+	live_table = dm_get_live_table(t->md);
+	if (live_table) {
+		if (dm_table_get_type(live_table) != dm_table_get_type(t)) {
+			DMWARN("can't change the device type after a table is bound");
+			dm_table_put(live_table);
+			return -EINVAL;
+		}
+		dm_table_put(live_table);
+	}
+
 	return 0;
 }
 
drivers/md/dm.c +0 -7
index a47f0b8..923b662 100644
@@ -2470,13 +2470,6 @@ struct dm_table *dm_swap_table(struct mapped_device *md, struct dm_table *table)
 		goto out;
 	}
 
-	/* cannot change the device type, once a table is bound */
-	if (md->map &&
-	    (dm_table_get_type(md->map) != dm_table_get_type(table))) {
-		DMWARN("can't change the device type after a table is bound");
-		goto out;
-	}
-
 	map = __bind(md, table, &limits);
 
 out:
#22 lore raw

PATCH v2 2/2 dm: only initialize full request_queue for request-based device

In reply to Mike Snitzer

Revert back to only allocating a minimalist request_queue structure initially (needed for both bio and request-based DM). Initialization of a full request_queue (request_fn, elevator, etc) is deferred until it is known that the DM device is request-based.

dm_init_request_based_queue is now called at the end of a request-based DM device's table load.

Otherwise bio-based DM devices will have an elevator, request_fn, etc. As a result bio-based DM devices had an 'iosched/' tree as well as a 'scheduler' other than "none" in sysfs.

---
 drivers/md/dm-table.c |    5 ++++
 drivers/md/dm.c       |   64 +++++++++++++++++++++++++++++++++++-------------
 drivers/md/dm.h       |    2 +
 3 files changed, 53 insertions(+), 18 deletions(-)
drivers/md/dm-table.c +5 -0
index 9924ea2..64a8578 100644
@@ -829,6 +829,11 @@ int dm_table_set_type(struct dm_table *t)
 		return -EINVAL;
 	}
 
+	if (!dm_init_request_based_queue(t->md)) {
+		DMWARN("Cannot initialize queue for Request-based dm");
+		return -EINVAL;
+	}
+
 	t->type = DM_TYPE_REQUEST_BASED;
 
 	return 0;
drivers/md/dm.c +46 -18

85 lines not on this page · raw message

drivers/md/dm.h +2 -0
index bad1724..489feba 100644
@@ -113,6 +113,8 @@ void dm_sysfs_exit(struct mapped_device *md);
 struct kobject *dm_kobject(struct mapped_device *md);
 struct mapped_device *dm_get_from_kobject(struct kobject *kobj);
 
+int dm_init_request_based_queue(struct mapped_device *md);
+
 /*
  * Targets for linear and striped mappings
  */
#23 lore raw

[RFC PATCH 3/2] dm: bio-based device must not register elevator in sysfs

Properly clear io scheduler state from sysfs if transitioning from a request-based table back to bio-based. Unregister the elevator from sysfs (removes 'iosched/') and have the 'scheduler' sysfs attribute properly report "none" for bio-based DM.

Adjust elv_iosched_show() to return "none" if !blk_queue_stackable. This allows the block layer to take bio-based DM into consideration.

These changes address the following (albeit obscure) corner-case:

   # dmsetup create --notable bio-based
   # echo "0 100 multipath ..." | dmsetup load bio-based
   # echo "0 100 linear ..."    | dmsetup load bio-based
   # dmsetup resume bio-based
   # ls /sys/block/dm-0/queue/
   ... iosched ...
---
 block/elevator.c      |    2 +-
 drivers/md/dm-table.c |    1 +
 drivers/md/dm.c       |   41 ++++++++++++++++++++++++++++++++++++++---
 drivers/md/dm.h       |    1 +
 4 files changed, 41 insertions(+), 4 deletions(-)
block/elevator.c +1 -1
index 6ec9137..168112e 100644
@@ -1088,7 +1088,7 @@ ssize_t elv_iosched_show(struct request_queue *q, char *name)
 	struct elevator_type *__e;
 	int len = 0;
 
-	if (!q->elevator)
+	if (!q->elevator || !blk_queue_stackable(q))
 		return sprintf(name, "none\n");
 
 	elv = e->elevator_type;
drivers/md/dm-table.c +1 -0
index 64a8578..dec8295 100644
@@ -803,6 +803,7 @@ int dm_table_set_type(struct dm_table *t)
 	if (bio_based) {
 		/* We must use this table as bio-based */
 		t->type = DM_TYPE_BIO_BASED;
+		dm_clear_request_based_queue(t->md);
 		return 0;
 	}
 
drivers/md/dm.c +38 -3

62 lines not on this page · raw message

drivers/md/dm.h +1 -0
index 489feba..95aec64 100644
@@ -114,6 +114,7 @@ struct kobject *dm_kobject(struct mapped_device *md);
 struct mapped_device *dm_get_from_kobject(struct kobject *kobj);
 
 int dm_init_request_based_queue(struct mapped_device *md);
+void dm_clear_request_based_queue(struct mapped_device *md);
 
 /*
  * Targets for linear and striped mappings
#24 lore raw

[PATCH 3/2 v2] dm: bio-based device must not register elevator in sysfs

Properly clear io scheduler state from sysfs if transitioning from a request-based table back to bio-based. Unregister the elevator from sysfs (removes 'iosched/') and have the 'scheduler' sysfs attribute properly report "none" for bio-based DM.

Adjust elv_iosched_show() to return "none" if !blk_queue_stackable. This allows the block layer to take bio-based DM into consideration.

These changes address the following (albeit obscure) corner-case:

   # dmsetup create --notable bio-based
   # echo "0 100 multipath ..." | dmsetup load bio-based
   # echo "0 100 linear ..."    | dmsetup load bio-based
   # dmsetup resume bio-based
   # ls /sys/block/dm-0/queue/
   ... iosched ...
---
 block/elevator.c      |    2 +-
 drivers/md/dm-table.c |    1 +
 drivers/md/dm.c       |   44 ++++++++++++++++++++++++++++++++++++--------
 drivers/md/dm.h       |    1 +
 4 files changed, 39 insertions(+), 9 deletions(-)

v2 changes:
* switched to using BUG_ON in dm_{init,clear}_request_based_queue
* refined dm_clear_request_based_queue slightly
* introduced dm_bio_based_md
* cleaned up some comments

drivers/md/dm-table.c +1 -0
Index: linux-2.6/drivers/md/dm-table.c
===================================================================
@@ -803,6 +803,7 @@ int dm_table_set_type(struct dm_table *t
 	if (bio_based) {
 		/* We must use this table as bio-based */
 		t->type = DM_TYPE_BIO_BASED;
+		dm_clear_request_based_queue(t->md);
 		return 0;
 	}
 
drivers/md/dm.c +36 -8

76 lines not on this page · raw message

drivers/md/dm.h +1 -0
Index: linux-2.6/drivers/md/dm.h
===================================================================
@@ -114,6 +114,7 @@ struct kobject *dm_kobject(struct mapped
 struct mapped_device *dm_get_from_kobject(struct kobject *kobj);
 
 int dm_init_request_based_queue(struct mapped_device *md);
+void dm_clear_request_based_queue(struct mapped_device *md);
 
 /*
  * Targets for linear and striped mappings
block/elevator.c +1 -1
Index: linux-2.6/block/elevator.c
===================================================================
@@ -1086,7 +1086,7 @@ ssize_t elv_iosched_show(struct request_
 	struct elevator_type *__e;
 	int len = 0;
 
-	if (!q->elevator)
+	if (!q->elevator || !blk_queue_stackable(q))
 		return sprintf(name, "none\n");
 
 	elv = e->elevator_type;
#25 lore raw
In reply to Mike Snitzer
Quoted 7 lines · Mike Snitzer

blk_init_queue() allocates the request_queue structure and then initializes it as needed (request_fn, elevator, etc).

Split initialization out to blk_init_allocated_queue_node. Introduce blk_init_allocated_queue wrapper function to model existing blk_init_queue and blk_init_queue_node interfaces.

Quoted 7 lines

Export elv_register_queue to allow a newly added elevator to be registered with sysfs. Export elv_unregister_queue for symmetry.

These changes allow DM to initialize a device's request_queue with more precision. In particular, DM no longer unconditionally initializes a full request_queue (elevator et al). It only does so for a request-based DM device.

Looks good, I'll apply this. I don't think the exports need to be _GPL in this case, generally I've only done that with the exports for hooking in a new IO scheduler.

Signature · 1 line
Jens Axboe
#26 lore raw
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Mike Snitzer

blk_init_queue() allocates the request_queue structure and then initializes it as needed (request_fn, elevator, etc).

Split initialization out to blk_init_allocated_queue_node. Introduce blk_init_allocated_queue wrapper function to model existing blk_init_queue and blk_init_queue_node interfaces.

Export elv_register_queue to allow a newly added elevator to be registered with sysfs. Export elv_unregister_queue for symmetry.

Quoted 8 lines
Quoted 4 lines

These changes allow DM to initialize a device's request_queue with more precision. In particular, DM no longer unconditionally initializes a full request_queue (elevator et al). It only does so for a request-based DM device.

Looks good, I'll apply this. I don't think the exports need to be _GPL in this case, generally I've only done that with the exports for hooking in a new IO scheduler.

OK, thanks for picking this up. Do you want me to send v2 that drops the _GPL or will you make those changes?

Mike

#27 lore raw
Quoted 19 lines · Mike Snitzer
Quoted 16 lines · Jens Axboe
Quoted 14 lines · Mike Snitzer

blk_init_queue() allocates the request_queue structure and then initializes it as needed (request_fn, elevator, etc).

Split initialization out to blk_init_allocated_queue_node. Introduce blk_init_allocated_queue wrapper function to model existing blk_init_queue and blk_init_queue_node interfaces.

Export elv_register_queue to allow a newly added elevator to be registered with sysfs. Export elv_unregister_queue for symmetry.

These changes allow DM to initialize a device's request_queue with more precision. In particular, DM no longer unconditionally initializes a full request_queue (elevator et al). It only does so for a request-based DM device.

Quoted 6 lines
Quoted 3 lines

Looks good, I'll apply this. I don't think the exports need to be _GPL in this case, generally I've only done that with the exports for hooking in a new IO scheduler.

OK, thanks for picking this up. Do you want me to send v2 that drops the _GPL or will you make those changes?

I just hand-edited that part. I can add the dm bit as well, when it has been reviewed and acked.

Signature · 1 line
Jens Axboe