Skip to contentSkip to conversation

copy on write misconception

DISCUSSION Mike Power 3 participants 6 messages Feb 22, 2013 – Feb 23, 2013 Btrfs file system

lore
Share & tools
lore

Mike Power

author
#0 lore raw

I think I have a misconception of what copy on write in btrfs means for individual files.

I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s

real 1m41.082s
user 0m0.000s
sys 0m7.792s

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

real 3m38.790s
user 0m0.124s
sys 0m10.709s

Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?

I was looking at a virtual machine solution and thought btrfs would be great if I could copy the vm disk to a new file at low cost and then launch that vm and customize it to my needs.

OS Ubuntu 12.10

Mike Power

#1 lore raw
Quoted 19 lines · Mike Power

I think I have a misconception of what copy on write in btrfs means for individual files.

I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s

real 1m41.082s
user 0m0.000s
sys 0m7.792s

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

Quoted 8 lines

real 3m38.790s
user 0m0.124s
sys 0m10.709s

Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?

You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).

Hugo.

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- "I don't like the look of it,  I tell you." "Well, stop ---     
                          looking at it, then."                          
author
#2 lore raw
Quoted 28 lines · Hugo Mills
Quoted 19 lines · Mike Power

I think I have a misconception of what copy on write in btrfs means for individual files.

I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s

real 1m41.082s
user 0m0.000s
sys 0m7.792s

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

Quoted 8 lines

real 3m38.790s
user 0m0.124s
sys 0m10.709s

Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?

Quoted 8 lines

You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).

Hugo.

I see, and it works great:
time cp --reflink=always 10G 10G3

real 0m0.028s
user 0m0.000s
sys 0m0.000s

So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.

In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.

Mike Power

#3 lore raw
Quoted 47 lines · Mike Power
Quoted 31 lines · Hugo Mills
Quoted 20 lines · Mike Power

I think I have a misconception of what copy on write in btrfs means for individual files.

I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s

real 1m41.082s
user 0m0.000s
sys 0m7.792s

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

Quoted 8 lines

real 3m38.790s
user 0m0.124s
sys 0m10.709s

Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?

Quoted 8 lines

You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).

Hugo.

I see, and it works great:
time cp --reflink=always 10G 10G3

real 0m0.028s
user 0m0.000s
sys 0m0.000s

Quoted 7 lines

So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.

In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.

There's no "copy a file" syscall; when a program copies a file, it opens a new file, and writes all the bytes from the old to the new. Converting this to a reflink would require btrfs to implement full de-dup (which is rather expensive), and still wouldn't prevent the program from reading and writing all 10gb (and so wouldn't be any faster).

You can set an alias in your shell to make cp --reflink=auto the default, but that won't affect other programs, nor other users.

author
#4 lore raw
Quoted 59 lines · cwillu
Quoted 44 lines · Mike Power
Quoted 28 lines · Hugo Mills
Quoted 19 lines · Mike Power

I think I have a misconception of what copy on write in btrfs means for individual files.

I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s

real 1m41.082s
user 0m0.000s
sys 0m7.792s

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

Quoted 8 lines

real 3m38.790s
user 0m0.124s
sys 0m10.709s

Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?

Quoted 8 lines

You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).

Hugo.

I see, and it works great:
time cp --reflink=always 10G 10G3

real 0m0.028s
user 0m0.000s
sys 0m0.000s

Quoted 7 lines

So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.

In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.

There's no "copy a file" syscall; when a program copies a file, it opens a new file, and writes all the bytes from the old to the new. Converting this to a reflink would require btrfs to implement full de-dup (which is rather expensive), and still wouldn't prevent the program from reading and writing all 10gb (and so wouldn't be any faster).

Quoted 2 lines

You can set an alias in your shell to make cp --reflink=auto the default, but that won't affect other programs, nor other users.

Thanks for the help guys. I learned that if I want some application to support this behavior they must specifically choose to implement it.

#5 lore raw
In reply to Mike Power
Quoted 5 lines

Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2

cp without arguments still does a regular copy; btrfs does nothing to de-duplicate writes.

"cp --reflink 10G 10G2" will give you the results you expect.