rbd: handle images flattened while mapped
Subsystems (2)
cover rbd: handle images flattened while mapped
This series detects when a mapped clone image gets flattened, and if any requests were in flight when that occurs, causes them to get resubmitted.
-Alex
[PATCH 1/4] rbd: detect when clone image is flattened [PATCH 2/4] rbd: re-submit read request for flattened clone [PATCH 3/4] rbd: re-submit write request for flattened clone [PATCH 4/4] rbd: re-submit flattened write request (part 2)
PATCH 1/4 rbd: detect when clone image is flattened
A format 2 clone image can be the subject of a "flatten" operation, during which all of its data gets "copied up" from its parent image, leaving the image fully populated. Once this is complete, the clone's association with the parent is abolished.
Since this can occur when a clone is mapped, we need to detect when it has occurred and handle it accordingly. We know an image has been flattened when we know it at one time had a parent, but we have learned (via a "get_parent" object class method call) it no longer has one.
There might be in-flight requests at the point we learn an image has been flattened, so we can't simply clean up parent data structures right away. Instead, we'll drop the initial parent reference when the parent has disappeared (rather than when the image gets destroyed), which will allow the last in-flight reference to clean things up when it's complete.
We leverage the fact that a zero parent overlap renders an image effectively unlayered. We set the overlap to 0 at the point we detect the clone image has flattened, which allows the unlayered behavior to take effect immediately, while keeping other parent structures in place until in-flight requests to complete.
This and the next few patches resolve:
http://tracker.ceph.com/issues/3763
--- drivers/block/rbd.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
drivers/block/rbd.c +0 -0
@@ -1927,6 +1927,11 @@ static void rbd_dev_parent_put(struct rbd_device
*rbd_dev)
* If an image has a non-zero parent overlap, get a reference to its
* parent.
*
+ * We must get the reference before checking for the overlap to
+ * coordinate properly with zeroing the parent overlap in
+ * rbd_dev_v2_parent_info() when an image gets flattened. We
+ * drop it again if there is no overlap.
+ *
* Returns true if the rbd device has a parent with a non-zero
* overlap and a reference for it was successfully taken, or
* false otherwise.
@@ -3771,8 +3776,26 @@ static int rbd_dev_v2_parent_info(struct rbd_device *rbd_dev)
end = reply_buf + ret;
ret = -ERANGE;
ceph_decode_64_safe(&p, end, parent_spec->pool_id, out_err);
- if (parent_spec->pool_id == CEPH_NOPOOL)
+ if (parent_spec->pool_id == CEPH_NOPOOL) {
+ /*
+ * Either the parent never existed, or we have
+ * record of it but the image got flattened so it no
+ * longer has a parent. When the parent of a
+ * layered image disappears we immediately set the
+ * overlap to 0. The effect of this is that all new
+ * requests will be treated as if the image had no
+ * parent.
+ */
+ if (rbd_dev->parent_overlap) {
+ rbd_dev->parent_overlap = 0;
+ smp_mb();
+ rbd_dev_parent_put(rbd_dev);
+ pr_info("%s: clone image has been flattened\n",
+ rbd_dev->disk->disk_name);
+ }
+
goto out; /* No parent? No problem. */
+ }
/* The ceph file layout needs to fit pool id in 32 bits */
@@ -4632,7 +4655,10 @@ static void rbd_dev_unprobe(struct rbd_device *rbd_dev)
{
struct rbd_image_header *header;
- rbd_dev_parent_put(rbd_dev);
+ /* Drop parent reference unless it's already been done (or none) */
+
+ if (rbd_dev->parent_overlap)
+ rbd_dev_parent_put(rbd_dev);
/* Free dynamic fields from the header, then zero it out */
Signature · 1 line
1.7.9.5
PATCH 2/4 rbd: re-submit read request for flattened clone
In reply to Alex ElderIf a clone image gets flattened while a parent read request is underway, the original rbd object request needs to be resubmitted.
The reason is that by the time we get the response to the parent read request, the data read from the parent may be out of date. In other words, we could see this sequence of events:
rbd client parent image/osd
---------- ----------------
original object ENOENT;
issue parent read
respond to parent read child image flattened
original image header refresh
<--- original object written independently here
parent read response received
Add code to rbd_img_parent_read_callback() to detect when a clone's parent image has disappeared (as evidenced by its parent overlap becoming 0), and re-submit the original read request in that case.
--- drivers/block/rbd.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)
drivers/block/rbd.c +0 -0
@@ -2672,14 +2672,36 @@ static void rbd_img_parent_read_callback(struct
rbd_img_request *img_request)
struct rbd_obj_request *obj_request;
struct rbd_device *rbd_dev;
u64 obj_end;
+ u64 img_xferred;
+ int img_result;
rbd_assert(img_request_child_test(img_request));
+ /* First get what we need from the image request and release it */
+
obj_request = img_request->obj_request;
+ img_xferred = img_request->xferred;
+ img_result = img_request->result;
+ rbd_img_request_put(img_request);
+
+ /*
+ * If the overlap has become 0 (most likely because the
+ * image has been flattened) we need to re-submit the
+ * original request.
+ */
rbd_assert(obj_request);
rbd_assert(obj_request->img_request);
+ rbd_dev = obj_request->img_request->rbd_dev;
+ if (!rbd_dev->parent_overlap) {
+ struct ceph_osd_client *osdc;
+
+ osdc = &rbd_dev->rbd_client->client->osdc;
+ img_result = rbd_obj_request_submit(osdc, obj_request);
+ if (!img_result)
+ return;
+ }
- obj_request->result = img_request->result;
+ obj_request->result = img_result;
if (obj_request->result)
goto out;
@@ -2692,7 +2714,6 @@ static void rbd_img_parent_read_callback(struct rbd_img_request *img_request)
*/
rbd_assert(obj_request->img_offset < U64_MAX - obj_request->length); obj_end = obj_request->img_offset + obj_request->length;
- rbd_dev = obj_request->img_request->rbd_dev;
if (obj_end > rbd_dev->parent_overlap) {
u64 xferred = 0;
@@ -2700,12 +2721,11 @@ static void rbd_img_parent_read_callback(struct rbd_img_request *img_request)
xferred = rbd_dev->parent_overlap -
obj_request->img_offset;
- obj_request->xferred = min(img_request->xferred, xferred);
+ obj_request->xferred = min(img_xferred, xferred);
} else {
- obj_request->xferred = img_request->xferred;
+ obj_request->xferred = img_xferred;
}
out:
- rbd_img_request_put(img_request);
rbd_img_obj_request_read_callback(obj_request); rbd_obj_request_complete(obj_request);
}
Signature · 1 line
1.7.9.5
PATCH 3/4 rbd: re-submit write request for flattened clone
In reply to Alex ElderAdd code to rbd_img_parent_read_full_callback() to detect when a clone's parent image has disappeared, and re-submit the original write request in that case. (See the previous commit for more reasoning about why this is appropriate.)
Rename some variables in rbd_img_obj_parent_read_full_callback() to match the convention used in the previous patch.
--- drivers/block/rbd.c | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-)
drivers/block/rbd.c +0 -0
@@ -2314,7 +2314,7 @@ rbd_img_obj_parent_read_full_callback(struct
rbd_img_request *img_request)
struct ceph_osd_client *osdc;
struct rbd_device *rbd_dev;
struct page **pages;
- int result;
+ int img_result;
u64 obj_size;
u64 offset;
u64 length;
@@ -2330,7 +2330,7 @@ rbd_img_obj_parent_read_full_callback(struct rbd_img_request *img_request)
orig_request = img_request->obj_request; rbd_assert(orig_request != NULL); rbd_assert(obj_request_type_valid(orig_request->type));
- result = img_request->result;
+ img_result = img_request->result;
obj_size = img_request->length; rbd_assert(obj_size == img_request->xferred); rbd_img_request_put(img_request);
@@ -2340,7 +2340,24 @@ rbd_img_obj_parent_read_full_callback(struct rbd_img_request *img_request)
rbd_assert(rbd_dev);
rbd_assert(obj_size == (u64)1 << rbd_dev->header.obj_order);
- if (result)
+ /*
+ * If the overlap has become 0 (most likely because the
+ * image has been flattened) we need to free the pages
+ * and re-submit the original write request.
+ */
+ if (!rbd_dev->parent_overlap) {
+ struct ceph_osd_client *osdc;
+ u32 page_count;
+
+ page_count = (u32)calc_pages_for(0, obj_size);
+ ceph_release_page_vector(pages, page_count);
+ osdc = &rbd_dev->rbd_client->client->osdc;
+ img_result = rbd_obj_request_submit(osdc, orig_request);
+ if (!img_result)
+ return;
+ }
+
+ if (img_result)
goto out_err;
/*
@@ -2349,7 +2366,7 @@ rbd_img_obj_parent_read_full_callback(struct rbd_img_request *img_request)
* request. Allocate the new copyup osd request for the
* original request, and release the old one.
*/
- result = -ENOMEM;
+ img_result = -ENOMEM;
osd_req = rbd_osd_req_create_copyup(orig_request); if (!osd_req)
goto out_err;
@@ -2383,13 +2400,13 @@ rbd_img_obj_parent_read_full_callback(struct rbd_img_request *img_request)
orig_request->callback = rbd_img_obj_copyup_callback; osdc = &rbd_dev->rbd_client->client->osdc;
- result = rbd_obj_request_submit(osdc, orig_request);
- if (!result)
+ img_result = rbd_obj_request_submit(osdc, orig_request);
+ if (!img_result)
return;
out_err:
/* Record the error code and complete the request */
- orig_request->result = result;
+ orig_request->result = img_result;
orig_request->xferred = 0;
obj_request_done_set(orig_request);
rbd_obj_request_complete(orig_request);
Signature · 1 line
1.7.9.5
PATCH 4/4 rbd: re-submit flattened write request (part 2)
In reply to Alex ElderAdd code to rbd_img_obj_exists_callback() to detect when a clone's parent image has disappeared, and re-submit the original write request in that case.
Kill off some redundant assertions.
This completes the resolution for:
http://tracker.ceph.com/issues/3763
--- drivers/block/rbd.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/block/rbd.c +0 -0
@@ -2508,6 +2508,7 @@ out_err: static void rbd_img_obj_exists_callback(struct rbd_obj_request
*obj_request)
{
struct rbd_obj_request *orig_request;
+ struct rbd_device *rbd_dev;
int result;
rbd_assert(!obj_request_img_data_test(obj_request));
@@ -2530,8 +2531,20 @@ static void rbd_img_obj_exists_callback(struct rbd_obj_request *obj_request)
obj_request->xferred, obj_request->length);
rbd_obj_request_put(obj_request);
- rbd_assert(orig_request);
- rbd_assert(orig_request->img_request);
+ /*
+ * If the overlap has become 0 (most likely because the
+ * image has been flattened) we need to free the pages
+ * and re-submit the original write request.
+ */
+ rbd_dev = orig_request->img_request->rbd_dev;
+ if (!rbd_dev->parent_overlap) {
+ struct ceph_osd_client *osdc;
+
+ osdc = &rbd_dev->rbd_client->client->osdc;
+ result = rbd_obj_request_submit(osdc, orig_request);
+ if (!result)
+ return;
+ }
/*
* Our only purpose here is to determine whether the object
Signature · 1 line
1.7.9.5
Quoted 10 lines · Alex Elder
This series detects when a mapped clone image gets flattened, and if any requests were in flight when that occurs, causes them to get resubmitted.
-Alex
[PATCH 1/4] rbd: detect when clone image is flattened [PATCH 2/4] rbd: re-submit read request for flattened clone [PATCH 3/4] rbd: re-submit write request for flattened clone [PATCH 4/4] rbd: re-submit flattened write request (part 2)
These all look good.