Online migration of arbitrary filesystems, possible?
Suppose that I have two big filesystems, dest fs and source fs.. I want to copy whole source fs to dest fs, but I need to keep source filesystem online I can't mount source fs read only and copy..
The idea is, mount both filesystems "together", and make write/read operations go on this way
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
dest fs
The problem here is, I need to remap every data and metadata on
step 3 (inodes and stuff), I think that read only data, isn't writed to disk,
so in case of old page, this page would be freed from memory and
not write to dest fs.. so I need read only pages to be forced dirty..
Write operations:
1. Write data to dest fs
2. Mark data as present on dest fs
Also I need to know when copy is finished, so I need a tool to crosscheck both filesystems and say what was copied and what wasn't, and give me some percent of remaining unsynced data. When I get 100% of data in sync I can umount source filesystem and use its storage for other propose.
The go here is move data online, where I can't bring data offline..
I'm a kernel newbie, I have just read a lot about page cache and VSF, but this seems possible, right?
Signature · 1 line
Follow the white rabbit!
Quoted 32 lines · Daniel Hilst
Suppose that I have two big filesystems, dest fs and source fs.. I want to copy whole source fs to dest fs, but I need to keep source filesystem online I can't mount source fs read only and copy..
The idea is, mount both filesystems "together", and make write/read operations go on this way
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
dest fsThe problem here is, I need to remap every data and metadata on
step 3 (inodes and stuff), I think that read only data, isn't writed to disk,
so in case of old page, this page would be freed from memory and
not write to dest fs.. so I need read only pages to be forced dirty..Write operations:
1. Write data to dest fs
2. Mark data as present on dest fsAlso I need to know when copy is finished, so I need a tool to crosscheck both filesystems and say what was copied and what wasn't, and give me some percent of remaining unsynced data. When I get 100% of data in sync I can umount source filesystem and use its storage for other propose.
Quoted 4 lines
The go here is move data online, where I can't bring data offline..
I'm a kernel newbie, I have just read a lot about page cache and VSF, but this seems possible, right?
Are you looking for a pre-existing solution? Or you want to write something?
Greg
Quoted 33 lines · Greg Freemyer
Quoted 32 lines · Daniel Hilst
Suppose that I have two big filesystems, dest fs and source fs.. I want to copy whole source fs to dest fs, but I need to keep source filesystem online I can't mount source fs read only and copy..
The idea is, mount both filesystems "together", and make write/read operations go on this way
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
dest fsThe problem here is, I need to remap every data and metadata on
step 3 (inodes and stuff), I think that read only data, isn't writed to disk,
so in case of old page, this page would be freed from memory and
not write to dest fs.. so I need read only pages to be forced dirty..Write operations:
1. Write data to dest fs
2. Mark data as present on dest fsAlso I need to know when copy is finished, so I need a tool to crosscheck both filesystems and say what was copied and what wasn't, and give me some percent of remaining unsynced data. When I get 100% of data in sync I can umount source filesystem and use its storage for other propose.
Quoted 9 lines
Quoted 4 lines
The go here is move data online, where I can't bring data offline..
I'm a kernel newbie, I have just read a lot about page cache and VSF, but this seems possible, right?
Are you looking for a pre-existing solution? Or you want to write something?
Greg
I can do this with btrfs snapshots, I think LVM2 can do this with snapshots too... I was searching for something like that, but using a kind of VFS hack
So I want to know if is possible, in theory, then I can start write something like that
Signature · 1 line
Follow the white rabbit!
Quoted 3 lines
Ive had to do something similar before. It is painful, but doable.
* create the directory structure on the target. (find -exec...)
* create a symlink for every file on the target device, obviously pointing to the origin device. (Find)
File system should be useable.
* progressively replace the symlinks with the actual file (find or script)
* watch out for actual symlinks...
Optionally replace the origin file with a symlink to the new file.
Signature · 9 lines
-- Teach your kids Science, or somebody else will :/ jason at ball.net vk2vjb at google.com <[email protected]> callsign: vk2vjb -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130402/c8603a58/attachment.html
Quoted 9 lines · Daniel Hilst
The idea is, mount both filesystems "together", and make write/read operations go on this way
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
Any reason you can't just 'rsync /source-fs /dest-fs'?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130329/56ab327c/attachment-0001.bin
Quoted 12 lines · Valdis.Kletnieks at vt.edu
Quoted 9 lines · Daniel Hilst
The idea is, mount both filesystems "together", and make write/read operations go on this way
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
Quoted 2 lines
Any reason you can't just 'rsync /source-fs /dest-fs'?
because I can't use dest-fs while rsynching
Signature · 1 line
Follow the white rabbit!
Quoted 3 lines · Daniel Hilst
Quoted 2 lines
Any reason you can't just 'rsync /source-fs /dest-fs'?
because I can't use dest-fs while rsynching
Sure you can. You just have to remember to pay attention to race conditions - if you create foo/bar.dat on the dest and then rsync wants to copy over a foo/bar.tar from the source, things will go poorly.
However, if you wanted to write to the dest while doing your sync, you'll have that issue no matter *what* method you use to do it.
Quoted 7 lines
Quoted 7 lines
Read operations:
1. See if data is already on dest fs,
2. If is then read data and bright back to caller (lets call this
cold read)
3. If is not, then read file from source fs, put it on page cache,
and change the backstorage of that page..
3.1 So when this page get dirty or too old, it will be writed to
You may want to look for 'overlayfs' and 'unionfs', which may provide you the function you need. (Note there's several different patchsets calling themselves 'unionfs').
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130402/778b299c/attachment.bin