Skip to contentSkip to conversation

rbd: clear EXISTS flag for disappearing snapshot

PATCH In review Alex Elder 2 participants 7 messages May 1, 2013 – May 2, 2013

Subsystems (2)
lore
Share & tools
lore

Alex Elder

author
#0 lore raw

cover rbd: clear EXISTS flag for disappearing snapshot

The first patch in this series puts back some functionality that was deleted inadvertently by the previous commit. Josh pointed out it had disappeared. The second one replaces a linear search of snapshot ids with a binary search for efficiency.

-Alex

[PATCH 1/2] rbd: clear EXISTS flag if mapped snapshot disappears [PATCH 2/2] rbd: use binary search for snapshot lookup

author
#1 lore raw

PATCH 1/2 rbd: clear EXISTS flag if mapped snapshot disappears

This functionality inadvertently disappeared in the last patch.

Image snapshots can get removed at just about any time. In particular it can disappear even if it is in use by an rbd client as a mapped image.

The rbd client deals with such a disappearance by responding to new requests with ENXIO. This is implemented by each rbd device maintaining an EXISTS flag, which is normally set but cleared if a snapshot disappears.

This patch (re-)implements the clearing of that flag.

Whenever mapped image header information is refreshed, if the mapping is for a snapshot, verify the mapped snapshot is still present in the updated snapshot context. If it is not, clear the flag.

It is not necessary to check this in the initial probe, because the probe will not succeed if the snapshot doesn't exist.

This resolves:

    http://tracker.ceph.com/issues/4880
---
 drivers/block/rbd.c |   23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/block/rbd.c +0 -0
index 0ca959f..3f58aba 100644
@@ -3114,6 +3114,25 @@ static int rbd_dev_v1_refresh(struct rbd_device

*rbd_dev)
return ret;
}

+/*
+ * Clear the rbd device's EXISTS flag if the snapshot it's mapped to
+ * has disappeared from the (just updated) snapshot context.
+ */
+static void rbd_exists_validate(struct rbd_device *rbd_dev) +{
+ u64 snap_id;
+
+ if (!test_bit(RBD_DEV_FLAG_EXISTS, &rbd_dev->flags))
+ return;
+
+ snap_id = rbd_dev->spec->snap_id;
+ if (snap_id == CEPH_NOSNAP)
+ return;
+
+ if (rbd_dev_snap_index(rbd_dev, snap_id) == BAD_SNAP_INDEX)
+ clear_bit(RBD_DEV_FLAG_EXISTS, &rbd_dev->flags);
+}
+
static int rbd_dev_refresh(struct rbd_device *rbd_dev)
{
u64 image_size;
@@ -3126,6 +3145,10 @@ static int rbd_dev_refresh(struct rbd_device *rbd_dev)
ret = rbd_dev_v1_refresh(rbd_dev);
else
ret = rbd_dev_v2_refresh(rbd_dev);
+
+ /* If it's a mapped snapshot, validate its EXISTS flag */
+
+ rbd_exists_validate(rbd_dev);
mutex_unlock(&ctl_mutex);
if (ret)
rbd_warn(rbd_dev, "got notification but failed to "

Signature · 1 line
1.7.9.5
author
#3 lore raw

PATCH 2/2 rbd: use binary search for snapshot lookup

In reply to Alex Elder

Use bsearch(3) to make snapshot lookup by id more efficient. (There could be thousands of snapshots, and conceivably many more.)

---
 drivers/block/rbd.c |   34 +++++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)
drivers/block/rbd.c +1 -0
index 3f58aba..a6e5fe3 100644
@@ -33,6 +33,7 @@
 #include <linux/ceph/mon_client.h>
 #include <linux/ceph/decode.h>
 #include <linux/parser.h>
+#include <linux/bsearch.h>
 
 #include <linux/kernel.h>
 #include <linux/device.h>
@@ -819,16 +820,39 @@ static const char *_rbd_dev_v1_snap_name(struct

rbd_device *rbd_dev, u32 which)
return kstrdup(snap_name, GFP_KERNEL);
}

+/*
+ * Snapshot id comparison function for use with qsort()/bsearch().
+ * Note that result is for snapshots in *descending* order.
+ */
+static int snapid_compare_reverse(const void *s1, const void *s2) +{
+ u64 snap_id1 = *(u64 *)s1;
+ u64 snap_id2 = *(u64 *)s2;
+
+ if (snap_id1 < snap_id2)
+ return 1;
+ return snap_id1 == snap_id2 ? 0 : 1;
+}
+
+/*
+ * Search a snapshot context to see if the given snapshot id is
+ * present.
+ *
+ * Returns the position of the snapshot id in the array if it's found,
+ * or BAD_SNAP_INDEX otherwise.
+ *
+ * Note: The snapshot array is in kept sorted (by the osd) in
+ * reverse order, highest snapshot id first.
+ */
static u32 rbd_dev_snap_index(struct rbd_device *rbd_dev, u64 snap_id) {
struct ceph_snap_context *snapc = rbd_dev->header.snapc;
- u32 which;
+ u64 *found;

- for (which = 0; which < snapc->num_snaps; which++)
- if (snapc->snaps[which] == snap_id)
- return which;
+ found = bsearch(&snap_id, &snapc->snaps, snapc->num_snaps,
+ sizeof (snap_id), snapid_compare_reverse);

- return BAD_SNAP_INDEX;
+ return found ? (u32)(found - &snapc->snaps[0]) : BAD_SNAP_INDEX;
}

static const char *rbd_dev_v1_snap_name(struct rbd_device *rbd_dev,

Signature · 1 line
1.7.9.5
author
#6 lore raw
In reply to Alex Elder
Quoted 7 lines · Alex Elder

The first patch in this series puts back some functionality that was deleted inadvertently by the previous commit. Josh pointed out it had disappeared. The second one replaces a linear search of snapshot ids with a binary search for efficiency.

-Alex

These patches are available in the "review/wip-slabs" branch of the ceph-client git repository. The patches in these other two series are there as well:

[PATCH 0/4] rbd: use slab caches for frequently allocated structures [PATCH 0/3] libceph: use slab caches in messenger and osd osd client

-Alex

Quoted 3 lines

[PATCH 1/2] rbd: clear EXISTS flag if mapped snapshot disappears [PATCH 2/2] rbd: use binary search for snapshot lookup

Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] rbd: clear EXISTS flag if mapped snapshot disappears

In reply to Alex Elder
Quoted 73 lines · Alex Elder

This functionality inadvertently disappeared in the last patch.

Image snapshots can get removed at just about any time. In particular it can disappear even if it is in use by an rbd client as a mapped image.

The rbd client deals with such a disappearance by responding to new requests with ENXIO. This is implemented by each rbd device maintaining an EXISTS flag, which is normally set but cleared if a snapshot disappears.

This patch (re-)implements the clearing of that flag.

Whenever mapped image header information is refreshed, if the mapping is for a snapshot, verify the mapped snapshot is still present in the updated snapshot context. If it is not, clear the flag.

It is not necessary to check this in the initial probe, because the probe will not succeed if the snapshot doesn't exist.

This resolves:

     http://tracker.ceph.com/issues/4880
---
  drivers/block/rbd.c |   23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)
drivers/block/rbd.c +19 -0
index 0ca959f..3f58aba 100644
@@ -3114,6 +3114,25 @@ static int rbd_dev_v1_refresh(struct rbd_device
 *rbd_dev)
  	return ret;
  }
 
+/*
+ * Clear the rbd device's EXISTS flag if the snapshot it's mapped to
+ * has disappeared from the (just updated) snapshot context.
+ */
+static void rbd_exists_validate(struct rbd_device *rbd_dev)
+{
+	u64 snap_id;
+
+	if (!test_bit(RBD_DEV_FLAG_EXISTS, &rbd_dev->flags))
+		return;
+
+	snap_id = rbd_dev->spec->snap_id;
+	if (snap_id == CEPH_NOSNAP)
+		return;
+
+	if (rbd_dev_snap_index(rbd_dev, snap_id) == BAD_SNAP_INDEX)
+		clear_bit(RBD_DEV_FLAG_EXISTS, &rbd_dev->flags);
+}
+
  static int rbd_dev_refresh(struct rbd_device *rbd_dev)
  {

u64 image_size;

drivers/block/rbd.c +4 -0
@@ -3126,6 +3145,10 @@ static int rbd_dev_refresh(struct rbd_device
 *rbd_dev)
  		ret = rbd_dev_v1_refresh(rbd_dev);
  	else
  		ret = rbd_dev_v2_refresh(rbd_dev);
+
+	/* If it's a mapped snapshot, validate its EXISTS flag */
+
+	rbd_exists_validate(rbd_dev);
  	mutex_unlock(&ctl_mutex);
  	if (ret)

rbd_warn(rbd_dev, "got notification but failed to "

Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] rbd: use binary search for snapshot lookup

In reply to Alex Elder
Quoted 26 lines · Alex Elder

Use bsearch(3) to make snapshot lookup by id more efficient. (There could be thousands of snapshots, and conceivably many more.)

---
  drivers/block/rbd.c |   34 +++++++++++++++++++++++++++++-----
  1 file changed, 29 insertions(+), 5 deletions(-)
drivers/block/rbd.c +2 -0
index 3f58aba..a6e5fe3 100644
@@ -33,6 +33,7 @@
  #include <linux/ceph/mon_client.h>
  #include <linux/ceph/decode.h>
  #include <linux/parser.h>
+#include <linux/bsearch.h>
 
  #include <linux/kernel.h>
  #include <linux/device.h>
@@ -819,16 +820,39 @@ static const char *_rbd_dev_v1_snap_name(struct
 rbd_device *rbd_dev, u32 which)
  	return kstrdup(snap_name, GFP_KERNEL);
  }
 
+/*

drivers/block/rbd.c static const char *_rbd_dev_v1_snap_name(struct · line 834

Quoted 10 lines
drivers/block/rbd.c +10 -0
+ * Snapshot id comparison function for use with qsort()/bsearch().
+ * Note that result is for snapshots in *descending* order.
+ */
+static int snapid_compare_reverse(const void *s1, const void *s2)
+{
+	u64 snap_id1 = *(u64 *)s1;
+	u64 snap_id2 = *(u64 *)s2;
+
+	if (snap_id1 < snap_id2)
+		return 1;

I think this 1 should be -1. Looks good otherwise.

Quoted 31 lines
drivers/block/rbd.c +17 -5
+	return snap_id1 == snap_id2 ? 0 : 1;
+}
+
+/*
+ * Search a snapshot context to see if the given snapshot id is
+ * present.
+ *
+ * Returns the position of the snapshot id in the array if it's found,
+ * or BAD_SNAP_INDEX otherwise.
+ *
+ * Note: The snapshot array is in kept sorted (by the osd) in
+ * reverse order, highest snapshot id first.
+ */
  static u32 rbd_dev_snap_index(struct rbd_device *rbd_dev, u64 snap_id)
  {
  	struct ceph_snap_context *snapc = rbd_dev->header.snapc;
-	u32 which;
+	u64 *found;
 
-	for (which = 0; which < snapc->num_snaps; which++)
-		if (snapc->snaps[which] == snap_id)
-			return which;
+	found = bsearch(&snap_id, &snapc->snaps, snapc->num_snaps,
+				sizeof (snap_id), snapid_compare_reverse);
 
-	return BAD_SNAP_INDEX;
+	return found ? (u32)(found - &snapc->snaps[0]) : BAD_SNAP_INDEX;
  }
 

static const char *rbd_dev_v1_snap_name(struct rbd_device *rbd_dev,

author
#5 lore raw

Re: [PATCH 2/2] rbd: use binary search for snapshot lookup

Quoted 26 lines · Josh Durgin
Quoted 25 lines · Alex Elder

Use bsearch(3) to make snapshot lookup by id more efficient. (There could be thousands of snapshots, and conceivably many more.)

---
  drivers/block/rbd.c |   34 +++++++++++++++++++++++++++++-----
  1 file changed, 29 insertions(+), 5 deletions(-)
drivers/block/rbd.c +1 -0
index 3f58aba..a6e5fe3 100644
@@ -33,6 +33,7 @@
  #include <linux/ceph/mon_client.h>
  #include <linux/ceph/decode.h>
  #include <linux/parser.h>
+#include <linux/bsearch.h>
 
  #include <linux/kernel.h>
  #include <linux/device.h>
@@ -819,16 +820,39 @@ static const char *_rbd_dev_v1_snap_name(struct
 rbd_device *rbd_dev, u32 which)
      return kstrdup(snap_name, GFP_KERNEL);
  }
 
Quoted 13 lines

drivers/block/rbd.c static const char *_rbd_dev_v1_snap_name(struct · line 834

Quoted 11 lines
drivers/block/rbd.c +11 -0
+/*
+ * Snapshot id comparison function for use with qsort()/bsearch().
+ * Note that result is for snapshots in *descending* order.
+ */
+static int snapid_compare_reverse(const void *s1, const void *s2)
+{
+    u64 snap_id1 = *(u64 *)s1;
+    u64 snap_id2 = *(u64 *)s2;
+
+    if (snap_id1 < snap_id2)
+        return 1;

I think this 1 should be -1. Looks good otherwise.

No, this one should be one (for reverse sort). But the other one--below--should be -1.

I tested this with a little user space program and I did in fact have it right at one time, but somehow I screwed it up when transferring it over.

Very good eye on these, as usual. Thanks a lot.

-Alex

Quoted 35 lines
Quoted 31 lines
drivers/block/rbd.c +17 -5
+    return snap_id1 == snap_id2 ? 0 : 1;
+}
+
+/*
+ * Search a snapshot context to see if the given snapshot id is
+ * present.
+ *
+ * Returns the position of the snapshot id in the array if it's found,
+ * or BAD_SNAP_INDEX otherwise.
+ *
+ * Note: The snapshot array is in kept sorted (by the osd) in
+ * reverse order, highest snapshot id first.
+ */
  static u32 rbd_dev_snap_index(struct rbd_device *rbd_dev, u64 snap_id)
  {
      struct ceph_snap_context *snapc = rbd_dev->header.snapc;
-    u32 which;
+    u64 *found;
 
-    for (which = 0; which < snapc->num_snaps; which++)
-        if (snapc->snaps[which] == snap_id)
-            return which;
+    found = bsearch(&snap_id, &snapc->snaps, snapc->num_snaps,
+                sizeof (snap_id), snapid_compare_reverse);
 
-    return BAD_SNAP_INDEX;
+    return found ? (u32)(found - &snapc->snaps[0]) : BAD_SNAP_INDEX;
  }
 

static const char *rbd_dev_v1_snap_name(struct rbd_device *rbd_dev,