Rename+crash behaviour of btrfs - nearly ext3!

Hi!

Following Ubuntu's dpkg+ext4 problems I wanted to see if btrfs would solve them all. And it nearly does! Now I wonder if the remaining 0.2 seconds window of exposing 0-size files could be closed too.

I tested using two simple scripts (attached for reference) on kernel 2.6.34-rc7:
- rentest creates files $i.tmp and renames to $i.cur,
- owtest does the same but overwrites existing $i.cur files, letting them run for 30-50 seconds then resetting the virtual machine.

The results for ext3 are as expected: 0-size files are never exposed as $i.cur, overwrites are atomic.

ext4 overwrites are /almost/ atomic (I get one 0-size file in owtest), lots of 0-size files are exposed in rentest (30 seconds window).

btrfs *nearly* does as well as ext3. Overwrites are atomic.

The rentest exposes only a 0.2 seconds windows of 0-size $i.cur files, so that a "ls --full-time" after the crash looks like this (notice the time between 01281.cur and 01292.tmp, only 0.2 seconds):
[...]

(quoted diff) +0 -5

-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.812016407 +0200 01280.cur
-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.835999490 +0200 01281.cur
-rw-r--r-- 1 root root  0 2010-05-17 17:06:25.868035485 +0200 01282.cur
[...]
-rw-r--r-- 1 root root  0 2010-05-17 17:06:26.080003626 +0200 01291.cur
-rw-rw-rw- 1 root root  0 2010-05-17 17:06:26.108010083 +0200 01292.tmp

Finally, xfs kills lots of existing files in owtest and exposes lots of 0-size files in rentest (both 40 seconds window).

If anybody is interested, the bunch of trimmed "ls --full-time" output for all filesystems is attached.

Thanks,
Jakob

Hi!

Following Ubuntu's dpkg+ext4 problems I wanted to see if btrfs would solve them all. And it nearly does! Now I wonder if the remaining 0.2 seconds window of exposing 0-size files could be closed too.

I tested using two simple scripts (attached for reference) on kernel 2.6.34-rc7:
- rentest creates files $i.tmp and renames to $i.cur,
- owtest does the same but overwrites existing $i.cur files, letting them run for 30-50 seconds then resetting the virtual machine.

The results for ext3 are as expected: 0-size files are never exposed as $i.cur, overwrites are atomic.

ext4 overwrites are /almost/ atomic (I get one 0-size file in owtest), lots of 0-size files are exposed in rentest (30 seconds window).

btrfs *nearly* does as well as ext3. Overwrites are atomic.

The rentest exposes only a 0.2 seconds windows of 0-size $i.cur files, so that a "ls --full-time" after the crash looks like this (notice the time between 01281.cur and 01292.tmp, only 0.2 seconds):
[...]

Quoted 30 lines · Jakob Unterwurzacher

Hi!

Following Ubuntu's dpkg+ext4 problems I wanted to see if btrfs would solve them all. And it nearly does! Now I wonder if the remaining 0.2 seconds window of exposing 0-size files could be closed too.

I tested using two simple scripts (attached for reference) on kernel 2.6.34-rc7:
- rentest creates files $i.tmp and renames to $i.cur,
- owtest does the same but overwrites existing $i.cur files, letting them run for 30-50 seconds then resetting the virtual machine.

The results for ext3 are as expected: 0-size files are never exposed as $i.cur, overwrites are atomic.

ext4 overwrites are /almost/ atomic (I get one 0-size file in owtest), lots of 0-size files are exposed in rentest (30 seconds window).

btrfs *nearly* does as well as ext3. Overwrites are atomic.

The rentest exposes only a 0.2 seconds windows of 0-size $i.cur files, so that a "ls --full-time" after the crash looks like this (notice the time between 01281.cur and 01292.tmp, only 0.2 seconds):
[...]

(quoted diff) +0 -5

-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.812016407 +0200 01280.cur
-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.835999490 +0200 01281.cur
-rw-r--r-- 1 root root  0 2010-05-17 17:06:25.868035485 +0200 01282.cur
[...]
-rw-r--r-- 1 root root  0 2010-05-17 17:06:26.080003626 +0200 01291.cur
-rw-rw-rw- 1 root root  0 2010-05-17 17:06:26.108010083 +0200 01292.tmp

Quoted 1 line

This isn't actually true. There is no window, the inode isn't written to disk until all of the data is flushed to disk. So the in memory inode will be update, and therefore show an i_size of 0 since the io hasn't finished, but if you were to crash at this point, when you came back up you'd have the old data in place because the new inode data wasn't written to disk. I have a feeling ext4 is the same way, but I'd have to check for sure. Thanks,

Quoted 6 lines

Quoted 6 lines

(quoted diff) +0 -5

-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.812016407 +0200 01280.cur
-rw-r--r-- 1 root root 20 2010-05-17 17:06:25.835999490 +0200 01281.cur
-rw-r--r-- 1 root root  0 2010-05-17 17:06:25.868035485 +0200 01282.cur
[...]
-rw-r--r-- 1 root root  0 2010-05-17 17:06:26.080003626 +0200 01291.cur
-rw-rw-rw- 1 root root  0 2010-05-17 17:06:26.108010083 +0200 01292.tmp

Quoted 8 lines

Quoted 1 line

This isn't actually true. There is no window, the inode isn't written to disk until all of the data is flushed to disk. So the in memory inode will be update, and therefore show an i_size of 0 since the io hasn't finished, but if you were to crash at this point, when you came back up you'd have the old data in place because the new inode data wasn't written to disk. I have a feeling ext4 is the same way, but I'd have to check for sure. Thanks,

Jacob, could you please confirm if your test includes a crash?

-chris

Hi!

Following Ubuntu's dpkg+ext4 problems I wanted to see if btrfs would solve them all. And it nearly does! Now I wonder if the remaining 0.2 seconds window of exposing 0-size files could be closed too.

That should be a zero second window, we try to force things to disk during renames.

Could you please try this patch:

fs/btrfs/ordered-data.c +0 -0

index c9f1020..9370a71 100644

Quoted 6 lines · Chris Mason

That should be a zero second window, we try to force things to disk during renames.

Could you please try this patch:

fs/btrfs/ordered-data.c int btrfs_add_ordered_operation(struct btrfs_trans_handle *trans, · line 809

Quoted 10 lines

fs/btrfs/ordered-data.c +1 -1

index c9f1020..9370a71 100644
@@ -806,7 +806,7 @@ int btrfs_add_ordered_operation(struct btrfs_trans_handle *trans,
 	 * if this file hasn't been changed since the last transaction
 	 * commit, we can safely return without doing anything
 	 */
-	if (last_mod < root->fs_info->last_trans_committed)
+	if (0 && last_mod < root->fs_info->last_trans_committed)

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Quoted 7 lines · Jakob Unterwurzacher

Quoted 6 lines · Chris Mason

That should be a zero second window, we try to force things to disk during renames.

Could you please try this patch:

Quoted 15 lines

fs/btrfs/ordered-data.c int btrfs_add_ordered_operation(struct btrfs_trans_handle *trans, · line 809

Quoted 10 lines

fs/btrfs/ordered-data.c +1 -1

index c9f1020..9370a71 100644
@@ -806,7 +806,7 @@ int btrfs_add_ordered_operation(struct btrfs_trans_handle *trans,
 	 * if this file hasn't been changed since the last transaction
 	 * commit, we can safely return without doing anything
 	 */
-	if (last_mod < root->fs_info->last_trans_committed)
+	if (0 && last_mod < root->fs_info->last_trans_committed)

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Thanks, so much for the easy fix. I'll take a look.

Quoted 5 lines

Quoted 3 lines

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Thanks, so much for the easy fix. I'll take a look.

Ohhhhh, I read your initial email wrong, I'm sorry. The test we're failing, the rentest, doesn't overwrite one file with another. It is just creating a file and then renaming it.

Btrfs is explicitly choosing not to sync the file in this case because the rename isn't replacing good old data with new unwritten data. The rename is taking new unwritten data and giving it a different name.

Are there applications that rely on this?

-chris

Quoted 9 lines · Chris Mason

Quoted 5 lines

Quoted 3 lines

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Thanks, so much for the easy fix. I'll take a look.

Ohhhhh, I read your initial email wrong, I'm sorry. The test we're failing, the rentest, doesn't overwrite one file with another. It is just creating a file and then renaming it.

Yes, the overwrite test goes perfectly fine.

Quoted 7 lines

Btrfs is explicitly choosing not to sync the file in this case because the rename isn't replacing good old data with new unwritten data. The rename is taking new unwritten data and giving it a different name.

Are there applications that rely on this?

-chris

Well, dpkg (the Debian/Ubuntu package manager) did. Then ext4 became the default fs in Ubuntu and massive breakage was reported [1]. Now dpkg is fsync()ing everything and is about 2x slower than it was with ext3 [2].

Btrfs is so close to getting it "right" that i wondered whether the new file name hitting the disk could be delayed that one second for the data to make it to disk first.

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=3D/dev/zero of=3Dfoo bs=3D1M count=3D1000
mv foo foo2

Should be expected to write 1GB of data.

* [Chris Mason]

Quoted 7 lines

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=/dev/zero of=foo bs=1M count=1000
mv foo foo2

Should be expected to write 1GB of data.

IIRC, the answer you're looking for is "it did with ext3 in the default data=ordered mode". Combine that with the ext3 data=ordered fsync() escalation where (again IIRC) fsync() tended to force a full sync() of the file system, and it's not that difficult to see why someone would program with the expectation above.

Anyway, there's still a question of if a new file system should emulate the quirks of the old file system (read: be bug compatible), or if you can just expect to be popular enough that userspace adapts to the new order and lets you do The Right Thing instead.

I'm more than open to discussion on this one, but I don't see how:

Should be expected to write 1GB of data.

The thing is that different apps have a different version of 'right'. Rename is atomically replacing one file with another, and I completely agree that when we have an established file on disk, we shouldn't replace it with something that is potentially garbage.

But for the zeros case we have a file that isn't on disk and we're just giving it a new name. I can see a different class of applications getting upset about renames slowing the system down dramatically because they suddenly imply a lot of IO.

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=/dev/zero of=foo bs=1M count=1000
mv foo foo2

Should be expected to write 1GB of data.

-chris

Quoted 17 lines · Chris Mason

The thing is that different apps have a different version of 'right'. Rename is atomically replacing one file with another, and I completely agree that when we have an established file on disk, we shouldn't replace it with something that is potentially garbage.

But for the zeros case we have a file that isn't on disk and we're just giving it a new name. I can see a different class of applications getting upset about renames slowing the system down dramatically because they suddenly imply a lot of IO.

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=/dev/zero of=foo bs=1M count=1000
mv foo foo2

Quoted 3 lines

Should be expected to write 1GB of data.

-chris

The idea would be to delay the rename hitting the disk until the data has been written anyway.
The mv would return immediately, and someday, after the data has been written to disk, the rename would be written to disk.

Quoted 5 lines

The idea would be to delay the rename hitting the disk until the data has been written anyway.
The mv would return immediately, and someday, after the data has been written to disk, the rename would be written to disk.

This is possible, but we have to choose between consuming unbounded resources while we queue up all the mvs or sometimes forcing the things to disk. At the end of the day, disks are so slow that eventually you do end up waiting on them.

-chris

Quoted 6 lines · Chris Mason

Quoted 5 lines

The idea would be to delay the rename hitting the disk until the data has been written anyway.
The mv would return immediately, and someday, after the data has been written to disk, the rename would be written to disk.

Quoted 7 lines

This is possible, but we have to choose between consuming unbounded resources while we queue up all the mvs or sometimes forcing the things to disk. At the end of the day, disks are so slow that eventually you do end up waiting on them.

-chris

I'm not sure how much memory a queued rename takes up, but the time that would be spent flushing it to disk would then be spent flushing file data, draining the write buffer and freeing memory, no?

That would be writing to disk

[Data..................][Rename] or
[Rename][Data..................]

Whether you drain the file data queue or the rename queue first, in the end you'd have to write it all....

I thought the problem of delaying the renames was complexity, well, at least T'Tso said it was [1] - I'm not sure if this applies to btrfs as well.

Quoted 1 line · Jakob Unterwurzacher

On 18/05/10 16:36, Chris Mason wrote:

Quoted 3 lines

I thought the problem of delaying the renames was complexity, well, at least T'Tso said it was [1] - I'm not sure if this applies to btrfs as

I'm afraid there are lots and lots of different issues at play. The most important way to look at it is that forcing data to disk is very slow, which is why we try to avoid it whenever we can.

Applications can request that the data go to disk via lots of different ways. Rename was never ever meant to be one of them, but it really does make sense to provide atomic replacement of old good data with new good data, so we've implemented that extra syncing.

Implementing syncing when userland doesn't expect extra syncing usually just make userland very unhappy. It's not that we can't do it it's that doing it has implications for every application that uses rename.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines

I'm not sure how much memory a queued rename takes up, but the time that would be spent flushing it to disk would then be spent flushing file data, draining the write buffer and freeing memory, no?

That would be writing to disk

[Data..................][Rename] or
[Rename][Data..................]

Actually it is:

[Data..................][allow the transaction commit to complete] or [allow the transaction commit to complete][Data..................]

The problem is that people think of the rename as a tiny thing, but it is really bundled in with all of the other metadata operations that were
done in the current transaction. The space that was allocated to hold
the new file name, the space that was freed to remove the old file name, the directory entries, the directory inode etc etc.

This means that holding back that one rename requires holding back every operation done to the filesystem.

In btrfs, we're still able to do fsyncs quickly in this case because we have a dedicated log for that. But there are a few different types of operations (like disk management) that require us to wait for the transaction to complete even when we use the dedicated log.

Quoted 3 lines

Whether you drain the file data queue or the rename queue first, in the end you'd have to write it all....

It's about latency. The latency required to write the entire file is unbounded (the size of the file is unbounded). The latency required to commit the transaction without the file data is bounded because we are able to control the amount of metadata in each transaction.

See the firefox vs ext3 wars for an example of all of this, it's the latency the firefox people were (rightly) complaining about.

Quoted 3 lines

I thought the problem of delaying the renames was complexity, well, at least T'Tso said it was [1] - I'm not sure if this applies to btrfs as well.

I'm afraid there are lots and lots of different issues at play. The most important way to look at it is that forcing data to disk is very slow, which is why we try to avoid it whenever we can.

Applications can request that the data go to disk via lots of different ways. Rename was never ever meant to be one of them, but it really does make sense to provide atomic replacement of old good data with new good data, so we've implemented that extra syncing.

Implementing syncing when userland doesn't expect extra syncing usually just make userland very unhappy. It's not that we can't do it it's that doing it has implications for every application that uses rename.

-chris

Quoted 23 lines · Jakob Unterwurzacher

Quoted 10 lines · Chris Mason

Quoted 6 lines

Quoted 4 lines

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Thanks, so much for the easy fix. I'll take a look.

Ohhhhh, I read your initial email wrong, I'm sorry. The test we're failing, the rentest, doesn't overwrite one file with another. It is just creating a file and then renaming it.

Yes, the overwrite test goes perfectly fine.

Quoted 8 lines

Btrfs is explicitly choosing not to sync the file in this case because the rename isn't replacing good old data with new unwritten data. The rename is taking new unwritten data and giving it a different name.

Are there applications that rely on this?

-chris

Quoted 9 lines

Well, dpkg (the Debian/Ubuntu package manager) did. Then ext4 became the default fs in Ubuntu and massive breakage was reported [1]. Now dpkg is fsync()ing everything and is about 2x slower than it was with ext3 [2].

Btrfs is so close to getting it "right" that i wondered whether the new file name hitting the disk could be delayed that one second for the data to make it to disk first.

The thing is that different apps have a different version of 'right'. Rename is atomically replacing one file with another, and I completely agree that when we have an established file on disk, we shouldn't replace it with something that is potentially garbage.

But for the zeros case we have a file that isn't on disk and we're just giving it a new name. I can see a different class of applications getting upset about renames slowing the system down dramatically because they suddenly imply a lot of IO.

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=/dev/zero of=foo bs=1M count=1000
mv foo foo2

Should be expected to write 1GB of data.

-chris

Just to weigh in here, I think that you have the right behaviour already. If an application wants to force this to sync the data to disk, it should use fsync() after the rename.

Quoted 16 lines · Jakob Unterwurzacher

Quoted 7 lines · Chris Mason

Quoted 4 lines

Quoted 3 lines

Ok, I upgraded to 2.6.34 final and switched to defconfig. I only did the rename test ( i.e. no overwrite ), the window is now 1.1s, both with vanilla and with the patch.

Thanks, so much for the easy fix. I'll take a look.

Ohhhhh, I read your initial email wrong, I'm sorry. The test we're failing, the rentest, doesn't overwrite one file with another. It is just creating a file and then renaming it.

Yes, the overwrite test goes perfectly fine.

Quoted 6 lines

Btrfs is explicitly choosing not to sync the file in this case because the rename isn't replacing good old data with new unwritten data. The rename is taking new unwritten data and giving it a different name.

Are there applications that rely on this?

Quoted 9 lines

Quoted 1 line

-chris

Well, dpkg (the Debian/Ubuntu package manager) did. Then ext4 became the default fs in Ubuntu and massive breakage was reported [1]. Now dpkg is fsync()ing everything and is about 2x slower than it was with ext3 [2].

Btrfs is so close to getting it "right" that i wondered whether the new file name hitting the disk could be delayed that one second for the data to make it to disk first.

The thing is that different apps have a different version of 'right'. Rename is atomically replacing one file with another, and I completely agree that when we have an established file on disk, we shouldn't replace it with something that is potentially garbage.

But for the zeros case we have a file that isn't on disk and we're just giving it a new name. I can see a different class of applications getting upset about renames slowing the system down dramatically because they suddenly imply a lot of IO.

I'm more than open to discussion on this one, but I don't see how:

rm -f foo2
dd if=/dev/zero of=foo bs=1M count=1000
mv foo foo2

Should be expected to write 1GB of data.