[Fwd: Re: Linking two files together][RFC]
Hi,
I hope that ideas about btrfs are not off-topic for this mailing list.
The forwarded message below was written by me on fedora-users. The thread is about the ability to link two files in a manner similar to "cat 1 2 >3 && rm 1 2" while avoiding any data movement on the disk.
The implementation should just put the original extents together in the new file. Is there any filesystem which is capable of doing that? As btrfs is already based on extents and COW, couldn't this feature be evaluated for feasibility? I think a lot of usages will be found for it if actually implemented.
Read the following part if interested.
Thanks.
-------- Original Message --------
From: - Thu May 27 20:44:26 2010
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Message-ID: <[email protected]>
Date: Thu, 27 May 2010 13:11:55 +0200
From: Roberto Ragusa <[email protected]>
User-Agent: Thunderbird 2.0.0.23 (X11/20090825)
MIME-Version: 1.0
To: Community support for Fedora users <[email protected]>
Subject: Re: Linking two files together
References: <7F593570D3366E4E85C76BAF70FD0EED0106DBF31FB1@CVMMBX.vetmed.wsu.edu> <[email protected]>
In-Reply-To: <[email protected]>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Quoted 9 lines · Kevin J. Cummings
Quoted 8 lines · David
Hello,
I have studied various filesystems, and am fairly familiar with how they are structured. However, I am currently stuck on trying to do what seems like a simple thing.
I would like to join two files together without having to physically copy bytes (i.e. I have vary large files, so I don't want to use 'cat'). It seems to me that it should be possible to simply modify the file entry in the filesystem such that the last inode of the first file points to the first inode of the second file. I guess this is similar to a "hard link", but used to join files rather than simply have another pointer to one file.
I have seen 'mmv' and 'lxsplit' and they all seem to do the same thing, namely they want to physically copy the bytes in order to join two files together.
Quoted 7 lines
Quoted 1 line
Is there any such utility in linux to perform such a hard link to join or connect two files together without having to copy bytes?
If you could guarantee that the last extent used by the first file was completely full of data with no extraneous bytes, it might be possible to "merge" the extent maps of the 2 files into a single file entry. If you cannot guarantee that, then you will have to copy bytes from the 2nd file to the end of the first file.
But everything becomes possible if the fileystem permits partially empty blocks in the middle of the file. No filesystem does it AFAIK, but it is not a big issue, as partial blocks (or compacted tails) are already permitted at the end of the file. New filesystems use extents rather than blocks, so if the extents are measured in bytes instead of 512b-blocks you can just use a smaller extent in the middle of the file where the join happened.
At this point, you can support inplace-joining, inplace-inflating (add 10000 bytes in this file at position 300000), inplace-erasure (remove 10000 bytes at position 300000) and data shuffling (swap the first 50meg of the file with the last 50meg).
With heavy usage you have just created a new kind of fragmentation, which can be corrected with the usual defragmentation tools (including "cp"). (add that fragmentation is losing importance with the spreading of SSD)
Considering that sparse files have been a reality for decades and that the implementation of operation with inside-file byte-grained extents is not more difficult than truncate, I wonder if we will see something of this kind in some advanced filesystem (btrfs?).
There are a lot of possible uses:
- delete/replace mail in mbox format repositories
- smart packaging (delete from tar, delete from zip)
- in-place iso creation
and.... just imagine.....
- video editing (!) add/remove/replace frames inside a 150GiB captured video
Where can you submit ideas to btrfs?
It also has COW, so everything becomes even more exciting...
Signature · 5 lines
Roberto Ragusa mail at robertoragusa.it -- Roberto Ragusa mail at robertoragusa.it
Quoted 3 lines · On Wednesday 09 June 2010 13:53:00 Roberto Ragusa wrote:
Hi,
=20
I hope that ideas about btrfs are not off-topic for this mailing list=
=2E
Quoted 8 lines
=20
The forwarded message below was written by me on fedora-users. The thread is about the ability to link two files in a manner similar to "cat 1 2 >3 && rm 1 2" while avoiding any data movement on the disk.
The implementation should just put the original extents together in the new file. Is there any filesystem which is capable of doing that? As btrfs is already based on extents and COW, couldn't this feature b=
e
Quoted 2 lines
evaluated for feasibility? I think a lot of usages will be found for it if actually implemented.
It will come naturally with online data deduplication -- though, at the=
moment=20
the only FS I know of that can do this is ZFS.
Otherwise, we would need a completely new system calls to perform those= =20
operations.
Quoted 16 lines
=20
Read the following part if interested.
=20
Thanks.
=20
-------- Original Message --------
From: - Thu May 27 20:44:26 2010
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Message-ID: <[email protected]>
Date: Thu, 27 May 2010 13:11:55 +0200
From: Roberto Ragusa <[email protected]>
User-Agent: Thunderbird 2.0.0.23 (X11/20090825)
MIME-Version: 1.0
To: Community support for Fedora users <[email protected]=
Quoted 14 lines
Subject: Re: Linking two files together
References:
<7F593570D3366E4E85C76BAF70FD0EED0106DBF31FB1@CVMMBX.vetmed.wsu.edu> <[email protected]> In-Reply-To:
<[email protected]>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=3DISO-8859-1
Content-Transfer-Encoding: 7bit
=20Quoted 4 lines · Kevin J. Cummings
Quoted 3 lines · David
Hello,
=20
I have studied various filesystems, and am fairly familiar with ho=
w they
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
are structured. However, I am currently stuck on trying to do what seems like a simple thing.
=20
I would like to join two files together without having to physical=
ly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
copy bytes (i.e. I have vary large files, so I don't want to use 'cat'). It seems to me that it should be possible to simply modify=
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
file entry in the filesystem such that the last inode of the first=
file
Quoted 1 line
Quoted 1 line
Quoted 1 line
points to the first inode of the second file. I guess this is simi=
lar
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
to a "hard link", but used to join files rather than simply have another pointer to one file.
=20
I have seen 'mmv' and 'lxsplit' and they all seem to do the same t=
hing,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
namely they want to physically copy the bytes in order to join two files together.
=20
Is there any such utility in linux to perform such a hard link to =
join
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
or connect two files together without having to copy bytes?
=20
If you could guarantee that the last extent used by the first file =
was
Quoted 1 line
Quoted 1 line
completely full of data with no extraneous bytes, it might be possi=
ble
Quoted 1 line
Quoted 1 line
to "merge" the extent maps of the 2 files into a single file entry.=
If
Quoted 1 line
Quoted 1 line
you cannot guarantee that, then you will have to copy bytes from th=
e 2nd
Quoted 3 lines
Quoted 1 line
file to the end of the first file.
=20
But everything becomes possible if the fileystem permits partially em=
pty
Quoted 1 line
blocks in the middle of the file. No filesystem does it AFAIK, but it=
is
Quoted 2 lines
not a big issue, as partial blocks (or compacted tails) are already permitted at the end of the file. New filesystems use extents rather =
than
Quoted 1 line
blocks, so if the extents are measured in bytes instead of 512b-block=
s you
Quoted 1 line
can just use a smaller extent in the middle of the file where the joi=
n
Quoted 3 lines
happened.
=20
At this point, you can support inplace-joining, inplace-inflating (ad=
d
Quoted 2 lines
10000 bytes in this file at position 300000), inplace-erasure (remove 10000 bytes at position 300000) and data shuffling (swap the first 50=
meg
Quoted 3 lines
of the file with the last 50meg).
=20
With heavy usage you have just created a new kind of fragmentation, w=
hich
Quoted 1 line
can be corrected with the usual defragmentation tools (including "cp"=
).
Quoted 1 line
(add that fragmentation is losing importance with the spreading of SS=
D)
Quoted 2 lines
=20
Considering that sparse files have been a reality for decades and tha=
t
Quoted 2 lines
the implementation of operation with inside-file byte-grained extents is not more difficult than truncate, I wonder if we will see somethin=
g
Quoted 8 lines
of this kind in some advanced filesystem (btrfs?).
=20
There are a lot of possible uses:
- delete/replace mail in mbox format repositories
- smart packaging (delete from tar, delete from zip)
- in-place iso creation
and.... just imagine.....
- video editing (!) add/remove/replace frames inside a 150GiB capture=
d
Quoted 4 lines
video
=20
Where can you submit ideas to btrfs?
It also has COW, so everything becomes even more exciting...
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=C3=B3w 30/85 tel. +48 (22) 646-61-51, 646-74-24 www.qbs.com.pl
System Zarz=C4=85dzania Jako=C5=9Bci=C4=85
zgodny z norm=C4=85 ISO 9001:2000
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Roberto Ragusa
I hope that ideas about btrfs are not off-topic for this mailing list.
The forwarded message below was written by me on fedora-users. The thread is about the ability to link two files in a manner similar to "cat 1 2 >3 && rm 1 2" while avoiding any data movement on the disk.
OCFS2 can do this today with "reflinks"
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 11 lines · Roberto Ragusa
I hope that ideas about btrfs are not off-topic for this mailing list.
The forwarded message below was written by me on fedora-users. The thread is about the ability to link two files in a manner similar to "cat 1 2 >3 && rm 1 2" while avoiding any data movement on the disk.
The implementation should just put the original extents together in the new file. Is there any filesystem which is capable of doing that? As btrfs is already based on extents and COW, couldn't this feature be evaluated for feasibility? I think a lot of usages will be found for it if actually implemented.
Btrfs already has a CLONE_RANGE ioctl that will clone a range of (block-aligned) bytes from file A to any offset in file B. The fs just fixes up the file metadata to reference the same bytes on disk without reading or writing any actual file data.
sage