Skip to contentSkip to conversation

fsx: add missing file size update on zero range operations

PATCH In review [email protected] 3 participants 10 messages Apr 8, 2020 – Apr 20, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 fsx: add missing file size update on zero range operations

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +2 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
 	}
 
 	end_offset = keep_size ? 0 : offset + length;
+	if (!keep_size && end_offset > file_size)
+		file_size = end_offset;
 
 	if (end_offset > biggest) {
 		biggest = end_offset;
Signature · 1 line
2.11.0
#1 lore raw
Quoted 21 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 5 lines
ltp/fsx.c +2 -0
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
 	}
 
 	end_offset = keep_size ? 0 : offset + length;
+	if (!keep_size && end_offset > file_size)
+		file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Brian

Quoted 6 lines
ltp/fsx.c +0 -0
 	if (end_offset > biggest) {
 		biggest = end_offset;
Signature · 1 line
2.11.0
#2 lore raw
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#3 lore raw
Quoted 33 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Quoted 3 lines

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \                               
                (len) = (size) - (off);         \

... where size is file_size. Hm?

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#4 lore raw
Quoted 39 lines · Brian Foster
Quoted 33 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Quoted 3 lines

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Quoted 6 lines

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \
                (len) = (size) - (off);         \

... where size is file_size. Hm?

That only updates the range's length, not the file_size.

Also, if you check the global style, you'll see that in the function for every operation that can change file size we do update file_size explicitly (e.g. do_preallocate(), and we call TRIM_OFF_LEN before calling it as well).

Thanks.

Quoted 17 lines

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#5 lore raw
Quoted 40 lines · Filipe Manana
Quoted 39 lines · Brian Foster
Quoted 36 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Quoted 9 lines
Quoted 6 lines

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \
                (len) = (size) - (off);         \

... where size is file_size. Hm?

That only updates the range's length, not the file_size.

Yes, but it caps the range to within file_size.

Quoted 5 lines

Also, if you check the global style, you'll see that in the function for every operation that can change file size we do update file_size explicitly (e.g. do_preallocate(), and we call TRIM_OFF_LEN before calling it as well).

do_preallocate() (fallocate) passes maxfilelen instead of file_size, as does write and mapwrite. Insert range uses TRIM_LEN() directly but also passes maxfilelen.

Brian

Quoted 20 lines

Thanks.

Quoted 17 lines

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#6 lore raw
Quoted 47 lines · Brian Foster
Quoted 45 lines · Filipe Manana
Quoted 44 lines · Brian Foster
Quoted 33 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Quoted 3 lines

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \
                (len) = (size) - (off);         \
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

... where size is file_size. Hm?

That only updates the range's length, not the file_size.

Yes, but it caps the range to within file_size.

Yes.

The problem I'm trying to solve is that because the file_size is not updated by zero range operations,
a following clone/dedupe/copy_range call will not be able to use a range that crosses the old file size and goes up to the new file size.

I.e. I'm not solving a problem where the range for those operations (or any others) incorrectly crosses eof - that doesn't happen because of the TRIM_* macros.

Does it make sense now?

Thanks.

Quoted 34 lines
Quoted 6 lines

Also, if you check the global style, you'll see that in the function for every operation that can change file size we do update file_size explicitly (e.g. do_preallocate(), and we call TRIM_OFF_LEN before calling it as well).

do_preallocate() (fallocate) passes maxfilelen instead of file_size, as does write and mapwrite. Insert range uses TRIM_LEN() directly but also passes maxfilelen.

Brian

Quoted 20 lines

Thanks.

Quoted 17 lines

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#7 lore raw
Quoted 62 lines · Filipe Manana
Quoted 47 lines · Brian Foster
Quoted 45 lines · Filipe Manana
Quoted 44 lines · Brian Foster
Quoted 33 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Quoted 3 lines

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \
                (len) = (size) - (off);         \
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

... where size is file_size. Hm?

That only updates the range's length, not the file_size.

Yes, but it caps the range to within file_size.

Yes.

The problem I'm trying to solve is that because the file_size is not updated by zero range operations,
a following clone/dedupe/copy_range call will not be able to use a range that crosses the old file size and goes up to the new file size.

Quoted 6 lines

I.e. I'm not solving a problem where the range for those operations (or any others) incorrectly crosses eof - that doesn't happen because of the TRIM_* macros.

Does it make sense now?

Not really. When is end_offset > file_size ever true in do_zero_range()?

Brian

Quoted 38 lines

Thanks.

Quoted 34 lines
Quoted 6 lines

Also, if you check the global style, you'll see that in the function for every operation that can change file size we do update file_size explicitly (e.g. do_preallocate(), and we call TRIM_OFF_LEN before calling it as well).

do_preallocate() (fallocate) passes maxfilelen instead of file_size, as does write and mapwrite. Insert range uses TRIM_LEN() directly but also passes maxfilelen.

Brian

Quoted 20 lines

Thanks.

Quoted 17 lines

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#8 lore raw
Quoted 64 lines · Brian Foster
Quoted 62 lines · Filipe Manana
Quoted 47 lines · Brian Foster
Quoted 45 lines · Filipe Manana
Quoted 44 lines · Brian Foster
Quoted 33 lines · Filipe Manana
Quoted 25 lines · Brian Foster
Quoted 23 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
      }
 
Quoted 6 lines

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 3 lines
ltp/fsx.c +2 -0
      end_offset = keep_size ? 0 : offset + length;
+     if (!keep_size && end_offset > file_size)
+             file_size = end_offset;

Should this ever happen if the caller uses TRIM_OFF_LEN() on the offset and length?

Quoted 3 lines

TRIM_OFF_LEN only trims the range, not the file_size. Or did I miss something?

Right, but TRIM_LEN() does:

        if ((off) + (len) > (size))             \
                (len) = (size) - (off);         \
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

... where size is file_size. Hm?

That only updates the range's length, not the file_size.

Yes, but it caps the range to within file_size.

Yes.

The problem I'm trying to solve is that because the file_size is not updated by zero range operations,
a following clone/dedupe/copy_range call will not be able to use a range that crosses the old file size and goes up to the new file size.

Quoted 8 lines
Quoted 6 lines

I.e. I'm not solving a problem where the range for those operations (or any others) incorrectly crosses eof - that doesn't happen because of the TRIM_* macros.

Does it make sense now?

Not really. When is end_offset > file_size ever true in do_zero_range()?

Ok, I see what you mean now. It's the trimming before calling the zero range function.
So we have a different type of problem - we never exercise zero range operations that cross the current eof.

Patch updated in the next verson.

Thanks.

Quoted 42 lines

Brian

Quoted 38 lines

Thanks.

Quoted 34 lines
Quoted 6 lines

Also, if you check the global style, you'll see that in the function for every operation that can change file size we do update file_size explicitly (e.g. do_preallocate(), and we call TRIM_OFF_LEN before calling it as well).

do_preallocate() (fallocate) passes maxfilelen instead of file_size, as does write and mapwrite. Insert range uses TRIM_LEN() directly but also passes maxfilelen.

Brian

Quoted 20 lines

Thanks.

Quoted 17 lines

Brian

Quoted 13 lines

Thanks.

Quoted 10 lines

Brian

Quoted 6 lines
ltp/fsx.c +0 -1
      if (end_offset > biggest) {
              biggest = end_offset;
--

2.11.0

#9 lore raw
In reply to [email protected]
Quoted 21 lines · On Wed, Apr 08, 2020 at 11:35:52AM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

When a zero range operation increases the size of the test file we were not updating the global variable 'file_size' which tracks the current size of the test file. This variable is used to for example compute the offset for a source range of clone, dedupe and copy file range operations.

So just fix it by updating the 'file_size' global variable whenever a zero range operation does not use the keep size flag and its range goes beyond the current file size.

---
 ltp/fsx.c | 2 ++
 1 file changed, 2 insertions(+)
ltp/fsx.c +0 -0
index 9d598a4f..fa383c94 100644

ltp/fsx.c do_zero_range(unsigned offset, unsigned length, int keep_size) · line 1216

Quoted 5 lines
ltp/fsx.c +2 -0
@@ -1212,6 +1212,8 @@ do_zero_range(unsigned offset, unsigned length, int keep_size)
 	}
 
 	end_offset = keep_size ? 0 : offset + length;
+	if (!keep_size && end_offset > file_size)
+		file_size = end_offset;

I think this should be done after we really excute fallocate(2), otherwise we may return early below:

if (testcalls <= simulatedopcount)
return;

Thanks,
Eryu

Quoted 6 lines
ltp/fsx.c +0 -0
 	if (end_offset > biggest) {
 		biggest = end_offset;
Signature · 1 line
2.11.0