btrfstune settings
Dear btrfs experts,
I hope this is the correct place to ask, the wiki and manpages did not help me on these questions.
BTRFS has gained extended inode refs, skinny metadata and no-holes quite a while ago and these are now the defaults for new mkfs.btrfs. btrfstune can activate those features.
However, I miss two things:
- How do I see on an existing FS which of these features are on?
btrfstune (it seems) can only "set", but not "get" the feature flags.
- Is it worthwhile / recommended / safe to activate those on existing FS?
Are there any steps (e.g. balancing metadata with -musage=0, I'd guess) needed to make them become active afterwards?
If there is any documentation on this and I missed it, please RTFM me.
Cheers and thanks a lot,
Oliver
Oliver Freyermuth posted on Sun, 28 Aug 2016 05:42:39 +0200 as excerpted:
Quoted 5 lines
Dear btrfs experts,
I hope this is the correct place to ask, the wiki and manpages did not help me on these questions.
Quoted 7 lines
BTRFS has gained extended inode refs, skinny metadata and no-holes quite a while ago and these are now the defaults for new mkfs.btrfs. btrfstune can activate those features.
However, I miss two things:
- How do I see on an existing FS which of these features are on? btrfstune (it seems) can only "set", but not "get" the feature flags.
Try btrfs-show-super <device>. The incompat_flags section enumerates the flags that are on (at least with a reasonably new btrfs-progs).
Quoted 4 lines
- Is it worthwhile / recommended / safe to activate those on existing FS?
Are there any steps (e.g. balancing metadata with -musage=0, I'd guess) needed to make them become active afterwards?
If they're intended to be changed on an existing filesystem, btrfstune should allow it, otherwise it doesn't. Node/leaf-size used to default to sectorsize (4K on x86/amd64), for instance, while now defaulting to 16K, but can't be changed on an existing filesystem, only at mkfs time, so that's the only place with the option.
In general it should be safe to activate flags via btrfstune, but since they'll generally apply only to newly written data, any benefits on a mature filesystem will be limited. For that reason as well as to get the benefit of 16K nodesize which you can't except at creation, I recommend backing up and recreating the filesystem from a fresh mkfs.btrfs.
Since btrfs is considered still stabilizing, having a backup is very strongly recommended in any case unless the value of the data simply isn't worth the hassle, and once you have a backup tested, available and ready to use should it be necessary, blowing the existing filesystem away and starting with a fresh filesystem created with the options you want isn't much more difficult and may well be faster than a full balance anyway. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Am Sun, 28 Aug 2016 08:27:59 +0000 (UTC)
schrieb Duncan <[email protected]>:
Quoted 7 lines
Quoted 4 lines
However, I miss two things:
- How do I see on an existing FS which of these features are on? btrfstune (it seems) can only "set", but not "get" the feature
flags.Try btrfs-show-super <device>. The
incompat_flagssection enumerates the flags that are on (at least with a reasonably new btrfs-progs).
It's also in the sub directories of /sys/fs/btrfs...
Signature · 4 lines
Regards, Kai Replies to list-only preferred.
(sorry if my Message-ID header is missing, I am not subscribed to the mailing list, so I reply using mail-archive)
Quoted 2 lines
Try btrfs-show-super <device>. The
incompat_flagssection enumerates the flags that are on (at least with a reasonably new btrfs-progs).
Thanks a lot for this hint - I totally missed that, since I only looked at manpages and features of the "all-in-one" btrfs tool up to now. Also thanks for the hint to /sys/fs/btrfs, this is even more readable!
And thanks a lot for your detailed answer(s) in general!
Quoted 5 lines
In general it should be safe to activate flags via btrfstune, but since they'll generally apply only to newly written data, any benefits on a mature filesystem will be limited. For that reason as well as to get the benefit of 16K nodesize which you can't except at creation, I recommend backing up and recreating the filesystem from a fresh mkfs.btrfs.
That would certainly be the best option, however,
I have two issues with that:
- Any replay of a backup will do a lot of writes on the SSD, reducing lifetime.
- I did not yet figure out a working way to get a "complete" backup for a btrfs. btrfs send is too slow for me, and rsync does an incomplete backup, since all FS-special attrs (like the NOCOW attr from 'chattr +C') are not copied, even when xattr copy is on.
My last backup replay was from an rsync backup (these I do regularly)
and afterwards I did a manual fixing of these special attrs, which is some bit of extra work. Also, the machine is offline during backup replay,
while I could still use it during a simple rebalance.
The good news is my old FS already has 16K leafsize, it's only missing skinny metadata (and no-holes, apparently that's not default in mkfs.btrfs yet?).
This reduces the scope of my question a bit - is it sufficient and worthwhile to "btrfstune -x" and then "btrfs balance start -musage=0 /" to convert all metadata, and thus gain some space?
Or are the gains not worth it and I should just wait until the next time I need to replay a backup for other reasons?
Cheers and thanks for your reply,
Oliver
Am Sun, 28 Aug 2016 18:18:52 +0200
schrieb Oliver Freyermuth <[email protected]>:
Quoted 21 lines
(sorry if my Message-ID header is missing, I am not subscribed to the mailing list, so I reply using mail-archive)
Quoted 3 lines
Try btrfs-show-super <device>. The
incompat_flagssection enumerates the flags that are on (at least with a reasonably new btrfs-progs).Thanks a lot for this hint - I totally missed that, since I only looked at manpages and features of the "all-in-one" btrfs tool up to now. Also thanks for the hint to
/sys/fs/btrfs, this is even more readable!And thanks a lot for your detailed answer(s) in general!
Quoted 6 lines
In general it should be safe to activate flags via btrfstune, but since they'll generally apply only to newly written data, any benefits on a mature filesystem will be limited. For that reason as well as to get the benefit of 16K nodesize which you can't except at creation, I recommend backing up and recreating the filesystem from a fresh mkfs.btrfs.
Quoted 8 lines
That would certainly be the best option, however,
I have two issues with that:
- Any replay of a backup will do a lot of writes on the SSD, reducing lifetime.
- I did not yet figure out a working way to get a "complete" backup for a btrfs. btrfs send is too slow for me, and rsync does an incomplete backup, since all FS-special attrs (like the NOCOW attr from 'chattr +C') are not copied, even when xattr copy is on.
Try borgbackup, I'm using it very successfully. It is very fast, supports very impressive deduplication and compression, retention policies, and remote backups - and it is available as a single binary version so you can more easily use it for disaster recovery. One
downside: while it seems to restore nocow attributes, it seems to do it in a silly way (it first creates the file, then sets the attributes, which of course won't work for nocow). I have not checked that extensively, only had to restore once yet.
Quoted 4 lines
My last backup replay was from an rsync backup (these I do regularly) and afterwards I did a manual fixing of these special attrs, which is some bit of extra work. Also, the machine is offline during backup replay, while I could still use it during a simple rebalance.
It's hard to circumvent this. I had once the idea to rsync backups to a btrfs device, then during recovery use it as a seeding device. I never tried it but it could work. Your system could be almost immediately back online this way but you'll have to do some special extra steps to get the seeding device back into a state which you can use again for your backups.
Quoted 3 lines
The good news is my old FS already has 16K leafsize, it's only missing skinny metadata (and no-holes, apparently that's not default in mkfs.btrfs yet?).
AFAIR no-holes and skinny-metadata can be enabled on existing filesystems...
Quoted 3 lines
This reduces the scope of my question a bit - is it sufficient and worthwhile to "btrfstune -x" and then "btrfs balance start -musage=0 /" to convert all metadata, and thus gain some space?
Not sure... Especially about no-holes. You could also try to "defragment" meta data using
$ find /mnt/btrfs-pool -type d -print0 | xargs -0 btrfs fi defrag
Quoted 2 lines
Or are the gains not worth it and I should just wait until the next time I need to replay a backup for other reasons?
It's probably not worth the hassle... Wait until you are forced to recreate the filesystem or you are having enough spare time to do it. However, some settings can be changed for existing filesystems and I would simply ignore the fact existing meta data needs to be converted.
Signature · 4 lines
Regards, Kai Replies to list-only preferred.
Quoted 8 lines
Try borgbackup, I'm using it very successfully. It is very fast, supports very impressive deduplication and compression, retention policies, and remote backups - and it is available as a single binary version so you can more easily use it for disaster recovery. One
downside: while it seems to restore nocow attributes, it seems to do it in a silly way (it first creates the file, then sets the attributes, which of course won't work for nocow). I have not checked that extensively, only had to restore once yet.
Wow - this looks like the holy grail I've been waiting for, not sure how I have missed that up to now.
Especially the deduplication across several backupped systems on the backup target is interesting, I originally planned to do that using duperemove on the backup target to dedupe across the readonly snapshots.
I'll certainly give borgbackup a spin as soon as I have time to look into it!
Quoted 4 lines
It's probably not worth the hassle... Wait until you are forced to recreate the filesystem or you are having enough spare time to do it. However, some settings can be changed for existing filesystems and I would simply ignore the fact existing meta data needs to be converted.
Thanks a lot for this hint and also your other points. I think I'll go that way, comparing a "similar" system with skinny metadata and one without the space usage for metadata seems not to be different by a noticeable factor.
Since my systems are SSD only, to avoid unnecessary writes I'll then just wait until I really need to replay a backup.
Thanks a lot and best regards,
Oliver
Hi,
happy borgbackup user here. This is probably off-topic for most but as many users probably are evaluating send/receive versus other backup solutions, I'll keep linux-btrfs in the loop.
Quoted 8 lines · Oliver Freyermuth
Quoted 8 lines
Try borgbackup, I'm using it very successfully. It is very fast, supports very impressive deduplication and compression, retention policies, and remote backups - and it is available as a single binary version so you can more easily use it for disaster recovery. One
downside: while it seems to restore nocow attributes, it seems to do it in a silly way (it first creates the file, then sets the attributes, which of course won't work for nocow). I have not checked that extensively, only had to restore once yet.
Quoted 5 lines
Wow - this looks like the holy grail I've been waiting for, not sure how I have missed that up to now.
Especially the deduplication across several backupped systems on the backup target is interesting, I originally planned to do that using duperemove on the backup target to dedupe across the readonly snapshots.
Note that only one backup can happen at a given time on a single repository. You'll probably have to both schedule the backups to avoid collisions and use "--lock-wait" with a large enough parameter to avoid backup failures.
There's another twist : borgbackup maintains a local index of the repositories' content, when it detects that it is out of sync (if several systems use the same repository it will) it has to update its index from remote.
I'm not sure how heavy this update can be (it seems it uses some form of delta). I have a ~user/.cache/borg of ~2GB for a user with ~9TB of data to backup in ~8M files but I don't share repositories so I'm not affected by this).
Best regards,
Lionel
Am Sun, 28 Aug 2016 20:10:38 +0200
schrieb Oliver Freyermuth <[email protected]>:
Quoted 9 lines
Quoted 8 lines
Try borgbackup, I'm using it very successfully. It is very fast, supports very impressive deduplication and compression, retention policies, and remote backups - and it is available as a single binary version so you can more easily use it for disaster recovery. One downside: while it seems to restore nocow attributes, it seems to do it in a silly way (it first creates the file, then sets the attributes, which of course won't work for nocow). I have not checked that extensively, only had to restore once yet.
Quoted 7 lines
Wow - this looks like the holy grail I've been waiting for, not sure how I have missed that up to now.
Especially the deduplication across several backupped systems on the backup target is interesting, I originally planned to do that using duperemove on the backup target to dedupe across the readonly snapshots. I'll certainly give borgbackup a spin as soon as I have time to look into it!
Apparently, only one client can access the repo at the same time. So you have to fiddle around with backup time windows and maybe the option for lock timeouts and retries via your service manager. I'm using it to backup a multi server shop system, it will first create an SQL dump, then start backing up each host one after the next to a local repository. In the end, I'll rsync it to an offsite location. It requires some scripting and ssh foo but it works reliable.
At home, I'm using a simple systemd job to wake my system up at night and put it back to sleep later:
$ systemctl cat internal-backup.{timer,service}
# /etc/systemd/system/internal-backup.timer
[Unit]
Description=Daily Backup Timer[Timer]
OnCalendar=03:00
WakeSystem=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/internal-backup.service [Unit] Description=Daily Backup Service
[Service] Environment=BORG_REPO=/mnt/private/backup/jupiter.sol.local.borg Environment=BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes Environment=BORG_RELOCATED_REPO_ACCESS_IS_OK=yes Type=oneshot
IOSchedulingClass=idle
IOSchedulingPriority=7
CPUSchedulingPolicy=batch
Nice=3
ProtectHome=read-only
ProtectSystem=full
PrivateTmp=yes
ReadWriteDirectories=/root/.cache/borg ReadOnlyDirectories=/mnt/btrfs-pool WorkingDirectory=/mnt/btrfs-pool ExecStart=/usr/bin/systemd-inhibit /usr/bin/borg create -v --stats --exclude-caches --compression zlib --exclude 'pp:gentoo/usr/portage/distfiles' --exclude 'pp:gentoo/rootfs/var/tmp' --exclude
'pp:gentoo/usr/portage/packages' ::'system@{now:%%Y%%m%%d-%%H%%M}' . ExecStartPost=/usr/bin/borg prune -v --prefix 'system@' --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --list --stats --save-space
The result then looks like this:
$ sudo -H borg
info /mnt/private/backup/jupiter.sol.local.borg::system@20160828-0255
Name: system@20160828-0255 Fingerprint:
786c9c8a18d5d04ba2a745d91777cf30345613b78d2afaa74192db871aa220de
Hostname: jupiter.sol.local Username: root
Time (start): Sun, 2016-08-28 02:55:24
Time (end): Sun, 2016-08-28 03:08:13
Command line: /usr/lib/python-exec/python3.4/borg create -v --stats
--exclude-caches --compression zlib --exclude
pp:gentoo/usr/portage/distfiles --exclude pp:gentoo/rootfs/var/tmp
--exclude pp:gentoo/usr/portage/packages ::system@{now:%Y%m%d-%H%M} .
Number of files: 2356372Original size Compressed size Deduplicated size
This archive: 1.73 TB 1.49 TB 1.02 GB All archives: 17.26 TB 14.94 TB 1.78 TB
Unique chunks Total chunks
Chunk index: 2858371 28473400
Initial backup creation is much slower than rsync but successive runs are much faster (around 15-20 minutes instead of 1 hour).
Signature · 4 lines
Regards, Kai Replies to list-only preferred.
Am Sun, 28 Aug 2016 18:18:52 +0200
schrieb Oliver Freyermuth <[email protected]>:
Quoted 4 lines
That would certainly be the best option, however, I have two issues with that:
- Any replay of a backup will do a lot of writes on the SSD, reducing lifetime.
I'm using bcache in front of my harddisks (writeback mode) to get almost-SSD performance. That way, when I restore, I can simply switch to writearound mode, restore, then switch back to writeback mode.
But in the long term, writeback mode will be extremely heavy on SSD wearing when the bcache is small because it will have to evict a lot of data from the cache. I'm using 500GB bcache for 3x 1TB harddisk. 500GB is more than my usual working set so bcache almost never has to evict data from the cache.
Signature · 4 lines
Regards, Kai Replies to list-only preferred.
Hi,
Quoted 7 lines · Oliver Freyermuth
I hope this is the correct place to ask, the wiki and manpages did not help me on these questions.
BTRFS has gained extended inode refs, skinny metadata and no-holes quite a while ago and these are now the defaults for new mkfs.btrfs. btrfstune can activate those features.
Quoted 8 lines
However, I miss two things:
- How do I see on an existing FS which of these features are on? btrfstune (it seems) can only "set", but not "get" the feature flags.
- Is it worthwhile / recommended / safe to activate those on existing FS? Are there any steps (e.g. balancing metadata with -musage=0, I'd guess) needed to make them become active afterwards?If there is any documentation on this and I missed it, please RTFM me.
all your questions should be now answered and documentation cross referenced among the relevant manual pages (will be synced to wiki at release time:
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/mkfs.btrfs.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfstune.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfs-man5.asciidoc
see sections 'FILESYSTEM FEATURES'.
Hi,
Am 01.09.2016 um 19:14 schrieb David Sterba:
Quoted 10 lines
all your questions should be now answered and documentation cross referenced among the relevant manual pages (will be synced to wiki at release time:
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/mkfs.btrfs.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfstune.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfs-man5.asciidocsee sections 'FILESYSTEM FEATURES'.
Thanks a lot for this documentation effort! Just to confirm, this indeed answers all my questions as a user.
I hope it will be helpful to many other btrfs-users in the future.
Best wishes,
Oliver