Skip to contentSkip to conversation

BTRFS RAID-1 leaf size change scenario

DISCUSSION Swâmi Petaramesh 3 participants 4 messages Feb 12, 2015 – Feb 13, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I have a BTRFS RAID-1 FS made from 2x 2TB SATA mechanical drives.

It was created a while ago, with defaults by the time of 4K leaf sizes.

It also contains *lots* of subvols and snapshots.

It has become very slow over time, and I know that BTRFS performs better with the new 16K leaf sizes.

I know that I cannot dynamically change the leaf size, but I'm wondering if the following could be feasible (and how to ?)

1/ Could I first pull a disk out of the current RAID-1 config, losing redundancy without breaking anything else ?

2/ Then reset the removed HD, and create onto it a new BTRFS FS with 16K leaf size ?

3/ Then is there a ways I could "btrfs send | btrfs receive" a complete volume including its subvolumes and snapshots,or is this impossible (and would I rather have to create the receiving volumes structure manually, and use rsync) ?

4/ Once the data are copied onto the new FS, could I reset the remaining "old" HD, import it into the new FS and get back to a RAID-1 config, rebuilding the RAID with a "balance" operation ?

(Machine's kernel is an Ubuntu 3.16.0-30 with btrfs-tools 3.14.1-1)

Many thanks for all help / lights about if this is feasible / how to do it without losing my data...

TIA.

Signature · 1 line
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
#1 lore raw
Quoted 1 line · Swâmi Petaramesh

It also contains *lots* of subvols and snapshots.

About how many is "lots"?

Quoted 7 lines

1/ Could I first pull a disk out of the current RAID-1 config, losing redundancy without breaking anything else ?

2/ Then reset the removed HD, and create onto it a new BTRFS FS with 16K leaf size ?

Quoted 8 lines

3/ Then is there a ways I could "btrfs send | btrfs receive" a complete volume including its subvolumes and snapshots,or is this impossible (and would I rather have to create the receiving volumes structure manually, and use rsync) ?

4/ Once the data are copied onto the new FS, could I reset the remaining "old" HD, import it into the new FS and get back to a RAID-1 config, rebuilding the RAID with a "balance" operation ?

You could do that, however at any point in the migration a read error/checksum mismatch could occur, and with the raid1 degraded, the entire point of having the btrfs raid1 in the first place is defeated.

Best practices suggests acquiring a 3rd drive to migrate the data to. Only once that's successful and confirmed, then you can obliterate one of the old raid1 mirrors, and put the other old mirror on a shelf JUST IN CASE. You can always mount it ro,degraded later.

When wiping one of the old mirrors, I go with some overkill by using btrfs-show-super -a to show all superblocks, and write 1MB of zeros to each super. Then add it to the new volume, then btrfs balance -dconvert=raid1 -mconvert=raid1.

The problems: All of your subvolumes and snapshots. To use btrfs send/receive on them, they each have to have a read-only version. And you have to have a naming convention that ensures you get either the -p or -c correct, so that you aren't unnecessarily duplicating data during the send/receive. If you don't get it right, you either miss migrating important data, or you'll run out of space on the destination. The same problem applies with rsync if you want to keep most or all of these snapshots.

The other option, is to make the raid1 volume a seed device, add two new drives, then delete the seed drive(s). I've only ever done this with a single device as seed, not a raid1. I don't even know if it will work because of this, but also there still may be seed device bugs in even recent kernels. The huge plus of this method though, is that you don't have to make a bunch of ro snapshots first, everything is migrated as it is on the seed. It's much easier. If it works. But since the seed is data and metadata raid1, so will be any added devices. So I think there isn't a way to make a raid1 a seed, where the added device is single profile. That'd be pretty nifty if it were possible.

Quoted 2 lines

(Machine's kernel is an Ubuntu 3.16.0-30 with btrfs-tools 3.14.1-1)

Unless this kernel contains, at a minimum, the btrfs fixes in 3.16.2, I would stop using it. There are also a set of fixes in 3.16.7 that ought to be used. Since 3.16 isn't even a listed longterm or stable kernel anymore, I suggest using 3.17.8, 3.18.3 or newer.

Quoted 2 lines

Many thanks for all help / lights about if this is feasible / how to do it without losing my data...

I think the strategy at this point necessitates a 3rd drive. And you're going to need to thin out the herd of subvols and snapshots you have to something that can be manageably migrated to the new volume. Once that's done, break one of the old mirrors to make it into a new mirror (conversion), and put the other old mirror on a shelf in case this whole thing goes badly. It's the only safe way.

Signature · 1 line
Chris Murphy
#2 lore raw

I'm going to amend what I wrote earlier. The problem with the seed device method, it won't let you change the leafsize. And that means you'll need to go with a new volume with mkfs, and migrate data with btrfs send receive instead.

And to clarify, you don't need to thin out subvolume to start out with on the existing volume. Do the thinning as part of the send/receive strategy. Keep the old raid1 mirror as a read only archive until you're clear you don't need it as much as you need a new backup volume. I wouldn't combine archive and backup in a single device.

Chris Murphy

#3 lore raw
In reply to Swâmi Petaramesh

Swâmi Petaramesh posted on Thu, 12 Feb 2015 14:26:09 +0100 as excerpted:

Quoted 8 lines

I have a BTRFS RAID-1 FS made from 2x 2TB SATA mechanical drives.

It was created a while ago, with defaults by the time of 4K leaf sizes.

It also contains *lots* of subvols and snapshots.

It has become very slow over time, and I know that BTRFS performs better with the new 16K leaf sizes.

I agree with everything Chris Murphy said and were it me, would do the migration as he suggested. Here I'll focus on another aspect, and reemphasize one he mentioned, as well.

1) Focus: Snapshots

Btrfs makes it deceivingly easy to make snapshots, since due to COW they can be created at very close to zero cost. Unfortunately, that ease of creation belies the much more complicated snapshot maintenance and deletion costs, and often people create and keep around far more snapshots than is healthy for an optimally functioning btrfs.

Basically, anything over say 500 snapshots on a btrfs is going to start bogging it down, and more than around 250-ish snapshots of any single subvolume should be entirely unnecessary. Unfortunately, due to the deceptive ease of creation, some people even take per-minute snapshots and fail to thin them down well over time, thus ending up with thousands to hundreds of thousands of snapshots, particularly if they're snapshotting multiple subvolumes at that extreme per-minute frequency. A filesystem in this condition is going to be a nightmare to do any reasonable maintenance (like a rebalance to add/remove/replace devices, or a defrag of more than a few files) on at all, and even regular operations will likely slow down due to fragmentation, etc.

Given your starred-emphasis "*lots*" of snapshots, I strongly suspect this to be one of the big reasons for your slowdowns, far more so than the 4k nodesize, tho that won't help. OTOH, if your characterization of *lots* was actually less than this, snapshotting probably isn't such a big problem after all and you can skip to the reemphasis point, below.

Unfortunately, at this point it may not be reasonable to recover from the situation on the existing filesystem, as doing the necessary thinning down of those snapshots could take nigh eternity (well, days per snapshot, "not reasonable" if you're dealing with anything near the thousands of snapshots I suspect) due to all that overhead.

But regardless of whether you can fix the existing btrfs, at least once you start over with a new one, try to better manage your snapshotting practices and I suspect the filesystem won't slow down as fast as this one did, while if you don't, I strongly suspect the newer 16k nodesizes aren't going to make that much difference and you'll get the same sort of slowdowns over time as you're dealing with now.

Here's the base argument concerning snapshot thinning management. Suppose you're doing hourly snapshots, and not doing any thinning. Suppose that a year later, you find you need a version of a file from a year ago, and go to retrieve it from one of those snapshots. So you go to mount a year-old snapshot and you have to pick one. Is it *REALLY* going to matter, a year on, with no reason to access it since, what exact hour it was? How are you even going to /know/ what exact hour to pick?

A year on, practicality suggests you'll simply pick one out of the 24 for the day and call it good. But is even that level of precision necessary? A year on, might a single snapshot for the week, or for the month, or even the quarter, be sufficient? Chances are it will be, and if the one you pick is too new or too old, you can simply pick one newer or one older and be done with it.

Similarly, per-minute snapshots? In the extreme case, maybe for a half our or hour. Then thin them down to say 10 minute snapshots, and to half hour snapshots after four or six hours (depending on whether you're basing on an 8-hour workday or a 24-hour day), then to hourly after a day, four or six hourly after three days, and daily after a week.

But in practice, per-minute snapshots are seldom necessary at all, and could be problems for maintenance if they end up taking more than a minute to delete. Ten minute, possibly, more likely half-hour or hourly is fine.

So say we start with half hour snapshots, 24-hours/day, but thinning down to hourly after four hours and to four-hourly after a day, for a week. That's:

8*half-hourly, 24-4=20*hourly, 7-1=6*6=36-4-hourly = 8+20+36 = 64 snapshots in a week.

Now, keep daily snapshots for three additional weeks = 64+21 = 85 snapshots in four weeks.

And keep weekly snapshots to fill out the half-year (26 weeks) 26-4 = 22 weeks, 22*7=154, 154+85 = 239 snapshots in a half a year.

Now after half a year, if the data is any value at all, it will have been backed up elsewhere. If you like, to avoid having to dig up those backups, you can keep quarterly snapshots for... pretty much the life of the filesystem or hardware, it'll only add four snapshots a year beyond the 239 for nearest half-year. Or delete snapshots beyond a quarter or a half year and rely on the off-filesystem backups, allowing btrfs to finally free the space tied up in the oldest and thus presumably most changed copies of the files in question.

As the above demonstrates, even at originally half-hourly snapshots, a reasonable thinning program keeps snapshots per subvolume to 200-300.

And if you can get by with say 4X per day (6-hourly on a 24-hour day) snapshots and keep only two days of that, thinning to 2X per day for a week, then daily for another week and weekly out to six months, that's:

2-days of 4X-daily = 8, 5 days of 2X = 10, 7 dailies, 24 weeklies = 8+10+7+24 = 49 snapshots to six months, starting with 6-hourly.

Now say two of the half-hourly snapshotted subvolumes and five of the 6- hourly snapshotted subvolumes. Rounding up for easier math and to allow for a few quarterly snapshots, we're now at 2*250=500 at the higher frequency, 5*60=300 at the lower frequency, 800 total for the filesystem.

800 snapshots for the filesystem is a bit high, but it's manageable, and **WELL** better than the tens or hundreds of thousands of snapshots that some are trying to handle. Obviously, if you can reasonably manage only one subvolume at the higher snapshot frequency, or if you can reduce snapshot frequency to daily or even not snapshot at all or only perhaps weekly for some of those subvolumes, total snapshots will go down accordingly.

Bottom line, if you're dealing with much over a thousand snapshots per filesystem, seriously reconsider your snapshotting strategy as you're probably doing it wrong. If possible, keep it to a few hundred. Filesystem management and even general usage should be far better as a result.

Again, while it might be too late to reasonably recover from a bad snapshotting strategy on the existing filesystem, as deleting all those old snapshots now may take far longer than is reasonable (tho you could try it and /see/ how long deleting a snapshot takes), at least try to manage things better when you setup the new filesystem.

2) Reemphasis: Versions.

Restating what CM said, old btrfs versions are buggy btrfs versions. Btrfs is still new enough and not yet stable and mature enough that running current versions really does lower the risk to your data, as old versions are known buggy versions, and running them is effectively playing Russian Roulette with your data -- you might get away with it for awhile, but play the odds long enough and eventually you'll get shot.

More specifically, at operational runtime, it's the kernel that's most vital, as userspace basically only tells the kernel what to do at a high level, and the kernel actually executes the lower level code to do it. So older kernels risk runtime damage on existing filesystems due to bugs that have long since been found and fixed in newer kernel versions.

However, in offline mode, such as when trying to repair an unmounted filesystem using btrfs check, or when using btrfs-restore to try to recover un-backed-up data from an unmountable filesystem before trying to repair it, btrfs-progs userspace becomes vital, as it's actually touching the filesystem then, without the kernel's direct involvement.

So a current kernel is most vital for btrfs at runtime, while a current btrfs-progs userspace is most vital if something screwed up and you're trying to fix it or recover what you can before blowing the existing filesystem away to start over.

Meanwhile, beyond data corruption bugs, one of the big focuses recently has been operation scaling. If you have thousands of snapshots as I suspect, it's very possible that the latest 3.18 or 3.19 kernel will actually let you work with them in a reasonable timeframe, while a 3.16 vintage kernel will take so long it's impractical to do anything with them at all. I'd certainly try it, at least, before giving up on doing anything practical with that old filesystem, unless of course you decide to simply bite the bullet and start over with a new filesystem and new devices, and only access the old one long enough to get current data and possibly a few selected snapshots off of it.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman