adding new devices to degraded raid1
I don't need to do it right away, but is it possible to add new devices to a degraded raid1?
One thing I might do in the future is replace a broken big drive with two small drives. It may even be used to migrate to SSDs.
Since btrfs-replace only seems to do 1:1 replacements, and I needed to physically remove an existing broken device to make room for the replacements, could I do something like:
mount -o degraded /mnt/foo
btrfs device add small1 small2 /mnt/foo
btrfs device remove broken /mnt/foo
?
Anyways, so far raid1 has been working great for me, but I have some devices nearing 70K Power_On_Hours according to SMART
Thanks.
Quoted 10 lines · Eric Wong
I don't need to do it right away, but is it possible to add new devices to a degraded raid1?
One thing I might do in the future is replace a broken big drive with two small drives. It may even be used to migrate to SSDs.
Since btrfs-replace only seems to do 1:1 replacements, and I needed to physically remove an existing broken device to make room for the replacements, could I do something like:
Quoted 6 lines
mount -o degraded
/mnt/foo
btrfs device add small1 small2/mnt/foo
btrfs device remove broken/mnt/foo?
Instead of
btrfs device remove broken /mnt/foo
You should do
btrfs device remove missing /mnt/foo
("missing" has to be write as is, it is a special term, see man page)
and
btrfs balance start /mnt/foo
To redistribute the data to the disks.
Please before trying it, wait for other suggestions or confirmation from more expert dveloper about that
BR
G.Baroncelli
Quoted 5 lines
Anyways, so far raid1 has been working great for me, but I have some devices nearing 70K
Power_On_Hoursaccording to SMARTThanks.
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 7 lines · Goffredo Baroncelli
Quoted 6 lines · Eric Wong
I don't need to do it right away, but is it possible to add new devices to a degraded raid1?
One thing I might do in the future is replace a broken big drive with two small drives. It may even be used to migrate to SSDs.
Quoted 7 lines
Quoted 7 lines
Since btrfs-replace only seems to do 1:1 replacements, and I needed to physically remove an existing broken device to make room for the replacements, could I do something like:
mount -o degraded
/mnt/foo
btrfs device add small1 small2/mnt/foo
btrfs device remove broken/mnt/foo
Note that add/remove is orders of magnitude slower than replace. Replace might take hours or even a day or two on a huge spinning drive. Add/remove might take _months_, though if you have 8-year-old disks then it's probably a few days, weeks at most.
Add/remove does work for raid1* (i.e. raid1, raid10, raid1c3, raid1c4). At the moment only 'replace' works reliably for raid5/raid6.
Quoted 9 lines
Quoted 2 lines
?
Instead of
btrfs device remove broken
/mnt/fooYou should do
Quoted 7 lines
btrfs device remove missing
/mnt/foo("missing" has to be write as is, it is a special term, see man page)
and
btrfs balance start
/mnt/foo
If the replacement disks are larger than half the size of the failed disk then device remove may do sufficient data relocation and you won't need balance. Once all the disks have equal amounts of unallocated space in 'btrfs fi usage' you can cancel any balances that are running.
On the other hand, if the replacement disks are close to half the size of the failed disk, then some careful balance filtering is required in order to utilize all the available space. This filtering is more than what the stock tool offers. You have to make sure that there are no block groups with a mirror copy on both of the small disks, as any such block group removes 1GB of available mirror space for data on the largest disk.
Quoted 17 lines
To redistribute the data to the disks.
Please before trying it, wait for other suggestions or confirmation from more expert dveloper about that
BR
G.BaroncelliQuoted 5 lines
Anyways, so far raid1 has been working great for me, but I have some devices nearing 70K
Power_On_Hoursaccording to SMARTThanks.
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 4 lines · Zygo Blaxell
Note that add/remove is orders of magnitude slower than replace. Replace might take hours or even a day or two on a huge spinning drive. Add/remove might take _months_, though if you have 8-year-old disks then it's probably a few days, weeks at most.
Btw, any explanation or profiling done on why remove is so much slower than replace? Especially since btrfs raid1 ought to be fairly mature at this point (and I run recent stable kernels).
Converting a single drive to raid1 was not slow at all, either. RAID 1 ought to be straightforward if there's plenty of free space, one would think...
Quoted 2 lines
Add/remove does work for raid1* (i.e. raid1, raid10, raid1c3, raid1c4). At the moment only 'replace' works reliably for raid5/raid6.
Noted, I'm staying far, far away from raid5/6 :) Thanks for your posts on that topic, by the way.
Quoted 10 lines · Goffredo Baroncelli
Quoted 9 lines
Instead of
btrfs device remove broken
/mnt/fooYou should do
btrfs device remove missing
/mnt/foo("missing" has to be write as is, it is a special term, see man page)
Thanks Goffredo, noted.
Quoted 9 lines
Quoted 3 lines
and
btrfs balance start
/mnt/fooIf the replacement disks are larger than half the size of the failed disk then device remove may do sufficient data relocation and you won't need balance. Once all the disks have equal amounts of unallocated space in 'btrfs fi usage' you can cancel any balances that are running.
Quoted 6 lines
On the other hand, if the replacement disks are close to half the size of the failed disk, then some careful balance filtering is required in order to utilize all the available space. This filtering is more than what the stock tool offers. You have to make sure that there are no block groups with a mirror copy on both of the small disks, as any such block group removes 1GB of available mirror space for data on the largest disk.
Yikes, that balancing sounds like a pain. I'm not super-limited on space, and a fair bit gets overwritten or replaced as time goes on, anyways.
I wonder how far I could get with some lossless rewrites which might make sense, anyways.
1) full "git gc" (I have a fair amount of git repos) Maybe setting pack.compression=0 will even help dedupe similar repos (but they'll be no fun to serve over network)
2) replacing some manually-compressed files with uncompressed versions (let btrfs compression handle it). I expect that'll let dedupe work better, too.
I have a lot of FLAC that could live as uncompressed .sox files. I expect FLAC to be more efficient on single files, but dedupe could save on cuts that are/were used for editing. I won't miss FLAC MD5 checksums when btrfs has checksums, either.
3) is this also something defrag can help with?
Thanks again.
Quoted 9 lines · Eric Wong
Quoted 4 lines · Zygo Blaxell
Note that add/remove is orders of magnitude slower than replace. Replace might take hours or even a day or two on a huge spinning drive. Add/remove might take _months_, though if you have 8-year-old disks then it's probably a few days, weeks at most.
Btw, any explanation or profiling done on why remove is so much slower than replace? Especially since btrfs raid1 ought to be fairly mature at this point (and I run recent stable kernels).
They do different things.
Replace just computes the contents of the filesystem the same way scrub
does: except for the occasional metadata seek, it runs at wire speeds because it reads blocks in order from one disk and writes in order on the other disk, 99.999% of the time.
Remove makes a copy of every extent, updates every reference to the extent, then deletes the original extents. Very seek-heavy--including seeks between reads and writes on the same drive--and the work is roughly proportional to the number of reflinks, so dedupe and snapshots push the cost up. About the only advantage of remove (and balance) is that it consists of 95% existing btrfs read and write code, and it can handle any relocation that does not require changing the size or content of an extent (including all possible conversions).
Arguably this isn't necessary. Remove could copy a complete block group, the same way replace does but to a different offset on each drive, and simply update the chunk tree with the new location of the block group at the end. Trouble is, nobody's implemented this approach in btrfs yet. It would be a whole new code path with its very own new bugs to fix.
Quoted 3 lines
Converting a single drive to raid1 was not slow at all, either. RAID 1 ought to be straightforward if there's plenty of free space, one would think...
Depends on the disk size, performance, and structure (how big the extents are and how many references). Also, "slow" is relative: 100x 2 minutes is not such a long time. 100x 20 hours is.
Quoted 39 lines
Quoted 2 lines
Add/remove does work for raid1* (i.e. raid1, raid10, raid1c3, raid1c4). At the moment only 'replace' works reliably for raid5/raid6.
Noted, I'm staying far, far away from raid5/6 :) Thanks for your posts on that topic, by the way.
Quoted 10 lines · Goffredo Baroncelli
Quoted 9 lines
Instead of
btrfs device remove broken
/mnt/fooYou should do
btrfs device remove missing
/mnt/foo("missing" has to be write as is, it is a special term, see man page)
Thanks Goffredo, noted.
Quoted 9 lines
Quoted 3 lines
and
btrfs balance start
/mnt/fooIf the replacement disks are larger than half the size of the failed disk then device remove may do sufficient data relocation and you won't need balance. Once all the disks have equal amounts of unallocated space in 'btrfs fi usage' you can cancel any balances that are running.
Quoted 6 lines
On the other hand, if the replacement disks are close to half the size of the failed disk, then some careful balance filtering is required in order to utilize all the available space. This filtering is more than what the stock tool offers. You have to make sure that there are no block groups with a mirror copy on both of the small disks, as any such block group removes 1GB of available mirror space for data on the largest disk.
Yikes, that balancing sounds like a pain. I'm not super-limited on space, and a fair bit gets overwritten or replaced as time goes on, anyways.
Quoted 6 lines
I wonder how far I could get with some lossless rewrites which might make sense, anyways.
1) full "git gc" (I have a fair amount of git repos) Maybe setting pack.compression=0 will even help dedupe similar repos (but they'll be no fun to serve over network)
Git pack doesn't do 4K block alignment, which limits filesystem-level dedupe opportunities. Git repos are strange: large ones are full of duplicate blocks, but only 3 or 4 at a time. By the time a big pack file has been cut up into extents that can be deduped, we've burned a gigabyte of IO, created 60 new extents out of 8, and might save 300K of space.
If you have a lot of related git repos, '.git/objects/info/alternates' is much more efficient than dedupe. Set up a repo that pulls refs/* to different remotes from all the other repos on the filesystem, and set all the other repos' alternates to point to the central repo. You'll only have each git object once on the filesystem after git gc. Aaaand you'll also have various issues with git auto-gc occasionally eating your reflogs. So maybe this is not for everyone.
Quoted 8 lines
2) replacing some manually-compressed files with uncompressed versions (let btrfs compression handle it). I expect that'll let dedupe work better, too.
I have a lot of FLAC that could live as uncompressed .sox files. I expect FLAC to be more efficient on single files, but dedupe could save on cuts that are/were used for editing. I won't miss FLAC MD5 checksums when btrfs has checksums, either.
If they're analog recordings (or have analog in any part of their mix) they will have nearly zero duplication. Dedupe only does bit-for-bit matches, and two clips that are off by one sample, or anything but an exact integer multiple of 1024 samples, will not be dedupeable. FLAC is much better than zstd.
VM image files compress and dedupe well. Better than xz if you have more than 2 or 3 big ones, but not as good as zpaq (which has its own deduper built-in, and it's more flexible than btrfs).
Quoted 1 line
3) is this also something defrag can help with?
Not really. defrag can make the balance run faster, but defrag will require almost the same amount of IO as the balance does. If you've already had to remove a disk, it's too late for defrag--it's something you have to maintain over time so that it's already done before a disk fails.
Quoted 2 lines
Thanks again.
28.08.2020 07:36, Zygo Blaxell пишет:
Quoted 6 lines
Replace just computes the contents of the filesystem the same way scrub
does: except for the occasional metadata seek, it runs at wire speeds because it reads blocks in order from one disk and writes in order on the other disk, 99.999% of the time.
Does it write them to the same absolute disk locations? IOW - is it possible to use smaller disk for replace or it must be at least as large as original disk?
Quoted 11 lines · Andrei Borzenkov
28.08.2020 07:36, Zygo Blaxell пишет:
Quoted 6 lines
Replace just computes the contents of the filesystem the same way scrub
does: except for the occasional metadata seek, it runs at wire speeds because it reads blocks in order from one disk and writes in order on the other disk, 99.999% of the time.Does it write them to the same absolute disk locations? IOW - is it possible to use smaller disk for replace or it must be at least as large as original disk?
Replace writes data to the locations recorded in the chunk tree, i.e. the original disk locations on the missing disk.
In theory, you can resize the offline disk to be smaller than the replacement disk, then run btrfs replace. In practice, only some of the methods work (e.g. you must specify device ID and not device name when replacing) and only on recent kernel versions.
btrfs dev remove is equivalent to 'btrfs fi resize <devid>:0' followed by "remove empty device <devid>" so the performance will be very similar for the portion of the data that is resized; however, a combination of resize and replace is still much faster than device remove, which does it the slow way for all of the data.
Quoted 11 lines · Zygo Blaxell
Quoted 9 lines · Eric Wong
Quoted 4 lines · Zygo Blaxell
Note that add/remove is orders of magnitude slower than replace. Replace might take hours or even a day or two on a huge spinning drive. Add/remove might take _months_, though if you have 8-year-old disks then it's probably a few days, weeks at most.
Btw, any explanation or profiling done on why remove is so much slower than replace? Especially since btrfs raid1 ought to be fairly mature at this point (and I run recent stable kernels).
Quoted 6 lines
They do different things.
Replace just computes the contents of the filesystem the same way scrub
does: except for the occasional metadata seek, it runs at wire speeds because it reads blocks in order from one disk and writes in order on the other disk, 99.999% of the time.
Thanks for the explanations. I'll heed your note down thread about doing a partial resize followed by a replace when possible.
Quoted 8 lines
Remove makes a copy of every extent, updates every reference to the extent, then deletes the original extents. Very seek-heavy--including seeks between reads and writes on the same drive--and the work is roughly proportional to the number of reflinks, so dedupe and snapshots push the cost up. About the only advantage of remove (and balance) is that it consists of 95% existing btrfs read and write code, and it can handle any relocation that does not require changing the size or content of an extent (including all possible conversions).
Does that mean remove speed would be closer to replace on good SSDs?
Quoted 5 lines
Arguably this isn't necessary. Remove could copy a complete block group, the same way replace does but to a different offset on each drive, and simply update the chunk tree with the new location of the block group at the end. Trouble is, nobody's implemented this approach in btrfs yet. It would be a whole new code path with its very own new bugs to fix.
Ah, it seems like a ton of work for a use case that mainly affects hobbyists. I won't hold my breath for it.
Quoted 7 lines
Quoted 3 lines
Converting a single drive to raid1 was not slow at all, either. RAID 1 ought to be straightforward if there's plenty of free space, one would think...
Depends on the disk size, performance, and structure (how big the extents are and how many references). Also, "slow" is relative: 100x 2 minutes is not such a long time. 100x 20 hours is.
It was a new, quickly filled FS; so probably unfragmented. I remember it seemed reasonable given the HW it was on.
Quoted 9 lines
Quoted 3 lines
1) full "git gc" (I have a fair amount of git repos) Maybe setting pack.compression=0 will even help dedupe similar repos (but they'll be no fun to serve over network)
Git pack doesn't do 4K block alignment, which limits filesystem-level dedupe opportunities. Git repos are strange: large ones are full of duplicate blocks, but only 3 or 4 at a time. By the time a big pack file has been cut up into extents that can be deduped, we've burned a gigabyte of IO, created 60 new extents out of 8, and might save 300K of space.
Heh. I'll just let git do its thing independently of btrfs. btrfs checksumming is great for ref storage, at least :>
Quoted 7 lines
If you have a lot of related git repos, '.git/objects/info/alternates' is much more efficient than dedupe. Set up a repo that pulls refs/* to different remotes from all the other repos on the filesystem, and set all the other repos' alternates to point to the central repo. You'll only have each git object once on the filesystem after git gc. Aaaand you'll also have various issues with git auto-gc occasionally eating your reflogs. So maybe this is not for everyone.
Yes, I've been using alternates with a mega repo for many years. I actually have all the remote fetch+url lines duplicated in the mega repo config for GC safety. It's a little more network traffic, but works with overwritten/throwaway branches in satellite repos.
<snip> will be sticking to FLAC as-is.
Quoted 3 lines
VM image files compress and dedupe well. Better than xz if you have more than 2 or 3 big ones, but not as good as zpaq (which has its own deduper built-in, and it's more flexible than btrfs).
Ah, it's a shame I needed to disable CoW on VM images to get acceptable performance, though. I'm using `bup' for backing up VMs and its a nice savings.
Quoted 6 lines
Quoted 1 line
3) is this also something defrag can help with?
Not really. defrag can make the balance run faster, but defrag will require almost the same amount of IO as the balance does. If you've already had to remove a disk, it's too late for defrag--it's something you have to maintain over time so that it's already done before a disk fails.
Alright, I'll make a note to keep things defragmented and avoid relying too much on reflinks.
Quoted 11 lines · Eric Wong
Quoted 8 lines · Zygo Blaxell
Remove makes a copy of every extent, updates every reference to the extent, then deletes the original extents. Very seek-heavy--including seeks between reads and writes on the same drive--and the work is roughly proportional to the number of reflinks, so dedupe and snapshots push the cost up. About the only advantage of remove (and balance) is that it consists of 95% existing btrfs read and write code, and it can handle any relocation that does not require changing the size or content of an extent (including all possible conversions).
Does that mean remove speed would be closer to replace on good SSDs?
It will be better, but there is still a cost for reading and writing non-contiguously. "Good SSD" depends on what the SSD is good at. A SSD rated for NAS or caching use would be OK, but a high-performance desktop SSD could hit big write-multiplication penalties. A couple of brand names starting with "S" have 5-second IO stalls when their internal caches get full. Proportionally, the ratio between the best and worst IO latency in these SSD models is as bad as SMR drives. Also there are CPU and IO latency costs for 'remove' in the host that don't go away no matter how good the disks are.
Quoted 8 lines
Quoted 5 lines
Arguably this isn't necessary. Remove could copy a complete block group, the same way replace does but to a different offset on each drive, and simply update the chunk tree with the new location of the block group at the end. Trouble is, nobody's implemented this approach in btrfs yet. It would be a whole new code path with its very own new bugs to fix.
Ah, it seems like a ton of work for a use case that mainly affects hobbyists. I won't hold my breath for it.
Well, by that argument, mdadm and lvm shouldn't be able to do it either, and yet they have supported this style of reshape for years.