copy on write misconception
I think I have a misconception of what copy on write in btrfs means for individual files.
I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/s
real 1m41.082s
user 0m0.000s
sys 0m7.792s
Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2
real 3m38.790s
user 0m0.124s
sys 0m10.709s
Oddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?
I was looking at a virtual machine solution and thought btrfs would be great if I could copy the vm disk to a new file at low cost and then launch that vm and customize it to my needs.
OS Ubuntu 12.10
Mike Power
Quoted 19 lines · Mike Power
I think I have a misconception of what copy on write in btrfs means for individual files.
I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/sreal 1m41.082s
user 0m0.000s
sys 0m7.792sThen if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2
Quoted 8 lines
real 3m38.790s
user 0m0.124s
sys 0m10.709sOddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?
You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- "I don't like the look of it, I tell you." "Well, stop ---
looking at it, then." Quoted 28 lines · Hugo Mills
Quoted 19 lines · Mike Power
I think I have a misconception of what copy on write in btrfs means for individual files.
I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/sreal 1m41.082s
user 0m0.000s
sys 0m7.792sThen if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2Quoted 8 lines
real 3m38.790s
user 0m0.124s
sys 0m10.709sOddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?
Quoted 8 lines
You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).Hugo.
I see, and it works great:
time cp --reflink=always 10G 10G3
real 0m0.028s
user 0m0.000s
sys 0m0.000s
So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.
In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.
Mike Power
Quoted 47 lines · Mike Power
Quoted 31 lines · Hugo Mills
Quoted 20 lines · Mike Power
I think I have a misconception of what copy on write in btrfs means for individual files.
I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/sreal 1m41.082s
user 0m0.000s
sys 0m7.792sThen if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2Quoted 8 lines
real 3m38.790s
user 0m0.124s
sys 0m10.709sOddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?
Quoted 8 lines
You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).Hugo.
I see, and it works great:
time cp --reflink=always 10G 10G3real 0m0.028s
user 0m0.000s
sys 0m0.000s
Quoted 7 lines
So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.
In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.
There's no "copy a file" syscall; when a program copies a file, it opens a new file, and writes all the bytes from the old to the new. Converting this to a reflink would require btrfs to implement full de-dup (which is rather expensive), and still wouldn't prevent the program from reading and writing all 10gb (and so wouldn't be any faster).
You can set an alias in your shell to make cp --reflink=auto the default, but that won't affect other programs, nor other users.
Quoted 59 lines · cwillu
Quoted 44 lines · Mike Power
Quoted 28 lines · Hugo Mills
Quoted 19 lines · Mike Power
I think I have a misconception of what copy on write in btrfs means for individual files.
I had originally thought that I could create a large file:
time dd if=/dev/zero of=10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 100.071 s, 107 MB/sreal 1m41.082s
user 0m0.000s
sys 0m7.792sThen if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2Quoted 8 lines
real 3m38.790s
user 0m0.124s
sys 0m10.709sOddly enough it actually takes longer then the initial file creation. So I am guessing that the long duration copy of the file is expected and that is not one of the virtues of btrfs copy on write. Does that sound right?
Quoted 8 lines
You probably want cp --reflink=always, which makes a CoW copy of
the file's metadata only. The resulting files have the semantics of two different files, but share their blocks until a part of one of them is modified (at which point, the modified blocks are no longer shared).Hugo.
I see, and it works great:
time cp --reflink=always 10G 10G3real 0m0.028s
user 0m0.000s
sys 0m0.000sQuoted 7 lines
So from the user perspective I might say I want to opt out of this feature not optin. I want all copies by all applications done as a copy on write. But if my understanding is correct that is up to the application being called (in this case cp) and how it in turns makes calls to the system.
In short I can't remount the btrfs filesystem with some new args that says always copy on write files because that is what it already.
There's no "copy a file" syscall; when a program copies a file, it opens a new file, and writes all the bytes from the old to the new. Converting this to a reflink would require btrfs to implement full de-dup (which is rather expensive), and still wouldn't prevent the program from reading and writing all 10gb (and so wouldn't be any faster).
Quoted 2 lines
You can set an alias in your shell to make cp --reflink=auto the default, but that won't affect other programs, nor other users.
Thanks for the help guys. I learned that if I want some application to support this behavior they must specifically choose to implement it.
Quoted 5 lines
Then if I copied this file no blocks would be copied until they are written. Hence the two files would use the same blocks underneath. But specifically that copy would be fast. Since it would only need to write some metadata. But when I copy the file:
time cp 10G 10G2
cp without arguments still does a regular copy; btrfs does nothing to de-duplicate writes.
"cp --reflink 10G 10G2" will give you the results you expect.