Skip to contentSkip to conversation

rbd: fix some issues around flushing notifies

PATCH In review Ilya Dryomov 2 participants 6 messages Mar 17, 2020

Subsystems (2)
lore
Share & tools
lore

Ilya Dryomov

author maintainer
#0 lore raw

cover rbd: fix some issues around flushing notifies

Hello,

A recent snapshot-based mirroring experiment exposed a deadlock on header_rwsem in the error path of rbd_dev_image_probe() (i.e. "rbd map").

Thanks,

Ilya

Ilya Dryomov (3):
rbd: avoid a deadlock on header_rwsem when flushing notifies
rbd: call rbd_dev_unprobe() after unwatching and flushing notifies
rbd: don't test rbd_dev->opts in rbd_dev_image_release()

 drivers/block/rbd.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
Signature · 1 line
2.19.2
author maintainer
#1 lore raw

PATCH 1/3 rbd: avoid a deadlock on header_rwsem when flushing notifies

rbd_unregister_watch() flushes notifies and therefore cannot be called under header_rwsem because a header update notify takes header_rwsem to synchronize with "rbd map". If mapping an image fails after the watch is established and a header update notify sneaks in, we deadlock when erroring out from rbd_dev_image_probe().

Move watch registration and unregistration out of the critical section. The only reason they were put there was to make header_rwsem management slightly more obvious.

Fixes 811c66887746 ("rbd: fix rbd map vs notify races")
---
 drivers/block/rbd.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/block/rbd.c +9 -4
index a4e7b494344c..f0ce30a6fc69 100644
@@ -4527,6 +4527,10 @@ static void cancel_tasks_sync(struct rbd_device *rbd_dev)
 	cancel_work_sync(&rbd_dev->unlock_work);
 }
 
+/*
+ * header_rwsem must not be held to avoid a deadlock with
+ * rbd_dev_refresh() when flushing notifies.
+ */
 static void rbd_unregister_watch(struct rbd_device *rbd_dev)
 {
 	cancel_tasks_sync(rbd_dev);
@@ -6907,6 +6911,8 @@ static void rbd_dev_image_release(struct rbd_device *rbd_dev)
  * device.  If this image is the one being mapped (i.e., not a
  * parent), initiate a watch on its header object before using that
  * object to get detailed information about the rbd image.
+ *
+ * On success, returns with header_rwsem held for write.
  */
 static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth)
 {
@@ -6936,6 +6942,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth)
 		}
 	}
 
+	down_write(&rbd_dev->header_rwsem);
 	ret = rbd_dev_header_info(rbd_dev);
 	if (ret) {
 		if (ret == -ENOENT && !need_watch)
@@ -6987,6 +6994,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth)
 err_out_probe:
 	rbd_dev_unprobe(rbd_dev);
 err_out_watch:
+	up_write(&rbd_dev->header_rwsem);
 	if (need_watch)
 		rbd_unregister_watch(rbd_dev);
 err_out_format:
@@ -7050,12 +7058,9 @@ static ssize_t do_rbd_add(struct bus_type *bus,
 		goto err_out_rbd_dev;
 	}
 
-	down_write(&rbd_dev->header_rwsem);
 	rc = rbd_dev_image_probe(rbd_dev, 0);
-	if (rc < 0) {
-		up_write(&rbd_dev->header_rwsem);
+	if (rc < 0)
 		goto err_out_rbd_dev;
-	}
 
 	if (rbd_dev->opts->alloc_size > rbd_dev->layout.object_size) {
 		rbd_warn(rbd_dev, "alloc_size adjusted to %u",
Signature · 1 line
2.19.2
author maintainer
#2 lore raw

PATCH 2/3 rbd: call rbd_dev_unprobe() after unwatching and flushing notifies

In reply to Ilya Dryomov

rbd_dev_unprobe() is supposed to undo most of rbd_dev_image_probe(), including rbd_dev_header_info(), which means that rbd_dev_header_info() isn't supposed to be called after rbd_dev_unprobe().

However, rbd_dev_image_release() calls rbd_dev_unprobe() before rbd_unregister_watch(). This is racy because a header update notify can sneak in:

"rbd unmap" thread ceph-watch-notify worker

rbd_dev_image_release()
rbd_dev_unprobe()
free and zero out header
rbd_watch_cb()
rbd_dev_refresh()
rbd_dev_header_info()
read in header

The same goes for "rbd map" because rbd_dev_image_probe() calls rbd_dev_unprobe() on errors. In both cases this results in a memory leak.

Fixes fd22aef8b47c ("rbd: move rbd_unregister_watch() call into rbd_dev_image_release()")
---
 drivers/block/rbd.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/block/rbd.c +4 -4
index f0ce30a6fc69..e590dc484c18 100644
@@ -6898,9 +6898,10 @@ static void rbd_print_dne(struct rbd_device *rbd_dev, bool is_snap)
 
 static void rbd_dev_image_release(struct rbd_device *rbd_dev)
 {
-	rbd_dev_unprobe(rbd_dev);
 	if (rbd_dev->opts)
 		rbd_unregister_watch(rbd_dev);
+
+	rbd_dev_unprobe(rbd_dev);
 	rbd_dev->image_format = 0;
 	kfree(rbd_dev->spec->image_id);
 	rbd_dev->spec->image_id = NULL;
@@ -6947,7 +6948,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth)
 	if (ret) {
 		if (ret == -ENOENT && !need_watch)
 			rbd_print_dne(rbd_dev, false);
-		goto err_out_watch;
+		goto err_out_probe;
 	}
 
 	/*
@@ -6992,11 +6993,10 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth)
 	return 0;
 
 err_out_probe:
-	rbd_dev_unprobe(rbd_dev);
-err_out_watch:
 	up_write(&rbd_dev->header_rwsem);
 	if (need_watch)
 		rbd_unregister_watch(rbd_dev);
+	rbd_dev_unprobe(rbd_dev);
 err_out_format:
 	rbd_dev->image_format = 0;
 	kfree(rbd_dev->spec->image_id);
Signature · 1 line
2.19.2
author maintainer
#3 lore raw

PATCH 3/3 rbd: don't test rbd_dev->opts in rbd_dev_image_release()

In reply to Ilya Dryomov

rbd_dev->opts is used to distinguish between the image that is being mapped and a parent. However, because we no longer establish watch for read-only mappings, this test is imprecise and results in unnecessary rbd_unregister_watch() calls.

Make it consistent with need_watch in rbd_dev_image_probe().

Fixes b9ef2b8858a0 ("rbd: don't establish watch for read-only mappings")
---
 drivers/block/rbd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/rbd.c +1 -1
index e590dc484c18..f44ce9ccadd6 100644
@@ -6898,7 +6898,7 @@ static void rbd_print_dne(struct rbd_device *rbd_dev, bool is_snap)
 
 static void rbd_dev_image_release(struct rbd_device *rbd_dev)
 {
-	if (rbd_dev->opts)
+	if (!rbd_is_ro(rbd_dev))
 		rbd_unregister_watch(rbd_dev);
 
 	rbd_dev_unprobe(rbd_dev);
Signature · 1 line
2.19.2
Reviewed-by on the v1 series
#4 lore raw
In reply to Ilya Dryomov
Quoted 17 lines · Ilya Dryomov

Hello,

A recent snapshot-based mirroring experiment exposed a deadlock on header_rwsem in the error path of rbd_dev_image_probe() (i.e. "rbd map").

Thanks,

Ilya

Ilya Dryomov (3):
rbd: avoid a deadlock on header_rwsem when flushing notifies
rbd: call rbd_dev_unprobe() after unwatching and flushing notifies
rbd: don't test rbd_dev->opts in rbd_dev_image_release()

Quoted 6 lines
 drivers/block/rbd.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

--
2.19.2

The "get_snapcontext" call still going to hang (albeit in an interruptible state) if the image has > 510 snapshots, correct?

Signature · 1 line
Jason
author maintainer
#5 lore raw
Quoted 22 lines · Jason Dillaman
Quoted 20 lines · Ilya Dryomov

Hello,

A recent snapshot-based mirroring experiment exposed a deadlock on header_rwsem in the error path of rbd_dev_image_probe() (i.e. "rbd map").

Thanks,

Ilya

Ilya Dryomov (3):
rbd: avoid a deadlock on header_rwsem when flushing notifies
rbd: call rbd_dev_unprobe() after unwatching and flushing notifies
rbd: don't test rbd_dev->opts in rbd_dev_image_release()

 drivers/block/rbd.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
Quoted 6 lines
Quoted 3 lines

--
2.19.2

The "get_snapcontext" call still going to hang (albeit in an interruptible state) if the image has > 510 snapshots, correct?

Yes, this has been a limitation of the messenger and its interface since day one. This is a krbd ticket specifically about snapshots, but this limitation affects a lot more than that:

https://tracker.ceph.com/issues/12874

It is engraved pretty deeply, but I'm planning to address it while the messenger is opened up for surgery.

Thanks,

Ilya