Skip to contentSkip to conversation

ceph: don't allow copy_file_range when stripe_count != 1

PATCH In review Luis Henriques 2 participants 4 messages Oct 31, 2019 Ceph client

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: don't allow copy_file_range when stripe_count != 1

copy_file_range tries to use the OSD 'copy-from' operation, which simply performs a full object copy. Unfortunately, the implementation of this system call assumes that stripe_count is always set to 1 and doesn't take into account that the data may be striped across an object set. If the file layout has stripe_count different from 1, then the destination file data will be corrupted.

For example:

Consider a 8 MiB file with 4 MiB object size, stripe_count of 2 and stripe_size of 2 MiB; the first half of the file will be filled with 'A's and the second half will be filled with 'B's:

               0      4M     8M       Obj1     Obj2
               +------+------+       +----+   +----+
        file:  | AAAA | BBBB |       | AA |   | AA |
               +------+------+       |----|   |----|
                                     | BB |   | BB |
                                     +----+   +----+

If we copy_file_range this file into a new file (which needs to have the same file layout!), then it will start by copying the object starting at file offset 0 (Obj1). And then it will copy the object starting at file offset 4M -- which is Obj1 again.

Unfortunately, the solution for this is to not allow remote object copies to be performed when the file layout stripe_count is not 1 and simply fallback to the default (VFS) copy_file_range implementation.

---
Hi Jeff,

I hope my understanding of the whole file striping in CephFS is correct; I had to go re-read the whole thing to refresh my memory.

Anyway, I guess that this is not really the only solution to this problem, but it's definitely the simplest one. copy_file_range is already way more complex that I had ever anticipated. I would rather keep this simple solution instead of adding more complexity and cover more corner cases. But yeah, we may want to revisit this in the future...

[OOT: files layout is probably one of the biggest headaches to sort out
the day we want to implement something like FIEMAP on CephFS ;-) ]

Cheers,
--
Luis

 fs/ceph/file.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/file.c +5 -2
index d277f71abe0b..3b0e6f9eb6a6 100644
@@ -1957,9 +1957,12 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
 		return -EOPNOTSUPP;
 
 	if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
-	    (src_ci->i_layout.stripe_count != dst_ci->i_layout.stripe_count) ||
-	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size))
+	    (src_ci->i_layout.stripe_count != 1) ||
+	    (dst_ci->i_layout.stripe_count != 1) ||
+	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size)) {
+		dout("Invalid src/dst files layout\n");
 		return -EOPNOTSUPP;
+	}
 
 	if (len < src_ci->i_layout.object_size)
 		return -EOPNOTSUPP; /* no remote copy will be done */
#1 lore raw
Quoted 62 lines · Luis Henriques

copy_file_range tries to use the OSD 'copy-from' operation, which simply performs a full object copy. Unfortunately, the implementation of this system call assumes that stripe_count is always set to 1 and doesn't take into account that the data may be striped across an object set. If the file layout has stripe_count different from 1, then the destination file data will be corrupted.

For example:

Consider a 8 MiB file with 4 MiB object size, stripe_count of 2 and stripe_size of 2 MiB; the first half of the file will be filled with 'A's and the second half will be filled with 'B's:

               0      4M     8M       Obj1     Obj2
               +------+------+       +----+   +----+
        file:  | AAAA | BBBB |       | AA |   | AA |
               +------+------+       |----|   |----|
                                     | BB |   | BB |
                                     +----+   +----+

If we copy_file_range this file into a new file (which needs to have the same file layout!), then it will start by copying the object starting at file offset 0 (Obj1). And then it will copy the object starting at file offset 4M -- which is Obj1 again.

Unfortunately, the solution for this is to not allow remote object copies to be performed when the file layout stripe_count is not 1 and simply fallback to the default (VFS) copy_file_range implementation.

---
Hi Jeff,

I hope my understanding of the whole file striping in CephFS is correct; I had to go re-read the whole thing to refresh my memory.

Anyway, I guess that this is not really the only solution to this problem, but it's definitely the simplest one. copy_file_range is already way more complex that I had ever anticipated. I would rather keep this simple solution instead of adding more complexity and cover more corner cases. But yeah, we may want to revisit this in the future...

[OOT: files layout is probably one of the biggest headaches to sort out
the day we want to implement something like FIEMAP on CephFS ;-) ]

Cheers,
--
Luis

 fs/ceph/file.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/file.c +0 -1
index d277f71abe0b..3b0e6f9eb6a6 100644
@@ -1957,9 +1957,12 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
 		return -EOPNOTSUPP;
 
 	if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
-	    (src_ci->i_layout.stripe_count != dst_ci->i_layout.stripe_count) ||

fs/ceph/file.c static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off, · line 1968

Quoted 10 lines
fs/ceph/file.c +5 -1
-	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size))
+	    (src_ci->i_layout.stripe_count != 1) ||
+	    (dst_ci->i_layout.stripe_count != 1) ||
+	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size)) {
+		dout("Invalid src/dst files layout\n");
 		return -EOPNOTSUPP;
+	}
 
 	if (len < src_ci->i_layout.object_size)
 		return -EOPNOTSUPP; /* no remote copy will be done */

I'm fine with restricting CFR to very simple cases, at least initially. We can always expand it later once the need becomes clear.

That said, we should probably add a comment explaining why we're excluding cases where the stripe count != 1 here. It doesn't need to contain the whole commit log message you wrote, but anyone that does want to improve this later might appreciate some breadcrumbs.

Maybe something like:

/*
* Striped file layouts require that we copy partial objects,
* but the OSD copy-from operation only supports full-object copies.
* Limit this to non-striped file layouts for now.
*/

If that sounds ok, I'll add that in and merge this later today.

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>
#2 lore raw
Quoted 84 lines · Jeff Layton
Quoted 62 lines · Luis Henriques

copy_file_range tries to use the OSD 'copy-from' operation, which simply performs a full object copy. Unfortunately, the implementation of this system call assumes that stripe_count is always set to 1 and doesn't take into account that the data may be striped across an object set. If the file layout has stripe_count different from 1, then the destination file data will be corrupted.

For example:

Consider a 8 MiB file with 4 MiB object size, stripe_count of 2 and stripe_size of 2 MiB; the first half of the file will be filled with 'A's and the second half will be filled with 'B's:

               0      4M     8M       Obj1     Obj2
               +------+------+       +----+   +----+
        file:  | AAAA | BBBB |       | AA |   | AA |
               +------+------+       |----|   |----|
                                     | BB |   | BB |
                                     +----+   +----+

If we copy_file_range this file into a new file (which needs to have the same file layout!), then it will start by copying the object starting at file offset 0 (Obj1). And then it will copy the object starting at file offset 4M -- which is Obj1 again.

Unfortunately, the solution for this is to not allow remote object copies to be performed when the file layout stripe_count is not 1 and simply fallback to the default (VFS) copy_file_range implementation.

---
Hi Jeff,

I hope my understanding of the whole file striping in CephFS is correct; I had to go re-read the whole thing to refresh my memory.

Anyway, I guess that this is not really the only solution to this problem, but it's definitely the simplest one. copy_file_range is already way more complex that I had ever anticipated. I would rather keep this simple solution instead of adding more complexity and cover more corner cases. But yeah, we may want to revisit this in the future...

[OOT: files layout is probably one of the biggest headaches to sort out
the day we want to implement something like FIEMAP on CephFS ;-) ]

Cheers,
--
Luis

 fs/ceph/file.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/file.c +0 -1
index d277f71abe0b..3b0e6f9eb6a6 100644
@@ -1957,9 +1957,12 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
 		return -EOPNOTSUPP;
 
 	if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
-	    (src_ci->i_layout.stripe_count != dst_ci->i_layout.stripe_count) ||

fs/ceph/file.c static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off, · line 1968

Quoted 10 lines
fs/ceph/file.c +5 -1
-	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size))
+	    (src_ci->i_layout.stripe_count != 1) ||
+	    (dst_ci->i_layout.stripe_count != 1) ||
+	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size)) {
+		dout("Invalid src/dst files layout\n");
 		return -EOPNOTSUPP;
+	}
 
 	if (len < src_ci->i_layout.object_size)
 		return -EOPNOTSUPP; /* no remote copy will be done */

I'm fine with restricting CFR to very simple cases, at least initially. We can always expand it later once the need becomes clear.

That said, we should probably add a comment explaining why we're excluding cases where the stripe count != 1 here. It doesn't need to contain the whole commit log message you wrote, but anyone that does want to improve this later might appreciate some breadcrumbs.

Maybe something like:

Quoted 7 lines

/*
* Striped file layouts require that we copy partial objects,
* but the OSD copy-from operation only supports full-object copies.
* Limit this to non-striped file layouts for now.
*/

If that sounds ok, I'll add that in and merge this later today.

Thanks, that looks good to me, feel free to add that comment.

Cheers,
--
Luís

#3 lore raw
Quoted 91 lines · Luis Henriques
Quoted 90 lines · Jeff Layton
Quoted 62 lines · Luis Henriques

copy_file_range tries to use the OSD 'copy-from' operation, which simply performs a full object copy. Unfortunately, the implementation of this system call assumes that stripe_count is always set to 1 and doesn't take into account that the data may be striped across an object set. If the file layout has stripe_count different from 1, then the destination file data will be corrupted.

For example:

Consider a 8 MiB file with 4 MiB object size, stripe_count of 2 and stripe_size of 2 MiB; the first half of the file will be filled with 'A's and the second half will be filled with 'B's:

               0      4M     8M       Obj1     Obj2
               +------+------+       +----+   +----+
        file:  | AAAA | BBBB |       | AA |   | AA |
               +------+------+       |----|   |----|
                                     | BB |   | BB |
                                     +----+   +----+

If we copy_file_range this file into a new file (which needs to have the same file layout!), then it will start by copying the object starting at file offset 0 (Obj1). And then it will copy the object starting at file offset 4M -- which is Obj1 again.

Unfortunately, the solution for this is to not allow remote object copies to be performed when the file layout stripe_count is not 1 and simply fallback to the default (VFS) copy_file_range implementation.

---
Hi Jeff,

I hope my understanding of the whole file striping in CephFS is correct; I had to go re-read the whole thing to refresh my memory.

Anyway, I guess that this is not really the only solution to this problem, but it's definitely the simplest one. copy_file_range is already way more complex that I had ever anticipated. I would rather keep this simple solution instead of adding more complexity and cover more corner cases. But yeah, we may want to revisit this in the future...

[OOT: files layout is probably one of the biggest headaches to sort out
the day we want to implement something like FIEMAP on CephFS ;-) ]

Cheers,
--
Luis

 fs/ceph/file.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/file.c +0 -1
index d277f71abe0b..3b0e6f9eb6a6 100644
@@ -1957,9 +1957,12 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
 		return -EOPNOTSUPP;
 
 	if ((src_ci->i_layout.stripe_unit != dst_ci->i_layout.stripe_unit) ||
-	    (src_ci->i_layout.stripe_count != dst_ci->i_layout.stripe_count) ||

fs/ceph/file.c static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off, · line 1968

Quoted 10 lines
fs/ceph/file.c +5 -1
-	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size))
+	    (src_ci->i_layout.stripe_count != 1) ||
+	    (dst_ci->i_layout.stripe_count != 1) ||
+	    (src_ci->i_layout.object_size != dst_ci->i_layout.object_size)) {
+		dout("Invalid src/dst files layout\n");
 		return -EOPNOTSUPP;
+	}
 
 	if (len < src_ci->i_layout.object_size)
 		return -EOPNOTSUPP; /* no remote copy will be done */

I'm fine with restricting CFR to very simple cases, at least initially. We can always expand it later once the need becomes clear.

That said, we should probably add a comment explaining why we're excluding cases where the stripe count != 1 here. It doesn't need to contain the whole commit log message you wrote, but anyone that does want to improve this later might appreciate some breadcrumbs.

Maybe something like:

/*
* Striped file layouts require that we copy partial objects,
* but the OSD copy-from operation only supports full-object copies.
* Limit this to non-striped file layouts for now.
*/

Quoted 4 lines
Quoted 1 line

If that sounds ok, I'll add that in and merge this later today.

Thanks, that looks good to me, feel free to add that comment.

Merged, and I also marked this for stable, since it's a potential data corruption bug.

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>