Skip to contentSkip to conversation

btrfs based backup?

DISCUSSION Ulli Horlacher 12 participants 31 messages Nov 12, 2019 – Mar 8, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

To have a secondary backup I will use btrfs send / receive,

Any comments on this? Or better suggestions?

The backup software must be open source!

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>
#1 lore raw

I highly recommend something similar to snapper or btrbk (https://github.com/digint/btrbk) for the automation of snapshotting.

I've used snapper previously and currently use btrbk, and both allow you to set very customizable retention policies for snapshots.

Say you take snapshots every hour, you could configure something like:
- Keep Hourly snapshots for 24 hours.
- Keep Daily snapshots for 7 days.
- Keep Weekly snapshots for 4 weeks.
- Keep Monthly snapshots for 6 months.

Of course you can optimize what snapshots you keep based on your knowledge of the data, and balancing point-in-time recovery vs not having too many snapshots to make some btrfs operations slower.

btrbk is focused towards running it both on a source and a destination server to automate send & receive for backup purposes, but it can also simply manage snapshots on the local machine.

Quoted 29 lines · Ulli Horlacher

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

To have a secondary backup I will use btrfs send / receive,

Any comments on this? Or better suggestions?

The backup software must be open source!

Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK 
Universitaet Stuttgart E-Mail: [email protected]
Allmandring 30a Tel: ++49-711-68565868
70569 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de
REF:<[email protected]>
#2 lore raw
In reply to Ulli Horlacher
Quoted 5 lines · Ulli Horlacher

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

My KISS script to do exactly this looks like so:
(Permissions on the backup are handled by default ACL's on the receiving end.)

/usr/bin/rsync \
	-a --no-o --no-g --no-p --chmod=ugo=rwX --info=STATS \
	--inplace --exclude='.snapshots' --delete \
	/nas/  \
	backup.server:/backups/server/nas/ \
    || exit 1

/usr/bin/ssh backup.server \
"/bin/btrfs file defrag -r -t 32M /backups/server"

/usr/bin/ssh backup.server \
"snapper -c server create -c number" \
|| exit 1

#3 lore raw
In reply to Ulli Horlacher

Hi,

I'm not sure if the btrfs list is the correct place for a generic answer - but I'll try to give one mentioning all the backup solutions I have collected experience with (all open source, of course).

1) btrbk ( https://github.com/digint/btrbk )
I use it on all my personal machines, both for local snapshotting (to unroll my own mistakes easily...) and for sending the incrementals
to an external storage. It's basically a well-working btrfs send / receive automation, so it needs btrfs at both ends (or becomes less efficient), which may not match your use case.

2) Borg Backup ( https://borgbackup.readthedocs.io/en/stable/ )
I use this whenever I do not have btrfs at one / both ends. It can also do encryption, compression and deduplication, purge old incrementals without ever doing a full backup, you can even mount your backups.
I use this for some smaller machines (e.g. on a Raspberry Pi) and we use it on our infrastructure for some configuration backups.

3) Restic ( https://restic.readthedocs.io/en/latest/ )
Restic is (feature-wise) like borg (but no compression yet). The main difference is that it can (but does not have to) back up to cloud-like storages such as S3. We intend to use this heavily to a local Ceph storage system with 3x replication offering S3/Swift via Rados Gateway nodes.
If you want something less heavy than a Ceph cluster (we love it, it does not bite!) you can try minio ( https://min.io/ ). I never used minio myself, but only heard good things about it.

4) Duplicati ( https://www.duplicati.com/ )
Like Borg / Restic (can also talk S3 if wanted, or store to a file system, also does compression).
The main advantage here is that it has a GUI. Probably not interesting for your use case, but we intend to recommend that to our users with Windows / MacOS X who may prefer some buttons to click.

5) Since you mention VMs in your signature, I'll also mention:
https://benji-backup.me/
http://backy2.com/
https://bitbucket.org/flyingcircus/backy
I'll personally recommend benji here, due to a large featureset, very active development and high efficiency. It does differential backups of RBD volumes, so it will only be really useful to you if you use Ceph RBD (you can also get it to work with LVM and raw block devices, I think).
You can find some of our experiences with it here:
https://indico.cern.ch/event/765214/contributions/3517132/

I think all of these are not too complex (of course, they only work well if your infrastructure matches them)
since you can essentially arrive at a working backup and restore in a few minutes.
I'll also add that for almost all of our servers, we do not do any backups at all - file servers and services with data have their storage replicated to their HA partner node(s), and all configuration is "backed up" by having it completely in Foreman / Puppet, so a machine can be reinstalled at the push of a button.

The main functionalities you give up with your original idea (rsync to one replicated node) is that you do not have deduplication built-in and would need to do encryption at the source yourself if needed. Also, you would have to do regular "full" backups and think about how you can keep incrementals - rsnapshot (which nowadays seems rather dead) could do something similar via rsync with hardlinks, but that meant you had tons of files which no FS really likes, and would always have to store a full file if a single byte changed.

Cheers and hope that helps,
Oliver

Am 12.11.19 um 19:34 schrieb Ulli Horlacher:

Quoted 25 lines

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

To have a secondary backup I will use btrfs send / receive,

Any comments on this? Or better suggestions?

The backup software must be open source!

#4 lore raw

freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

Cheers,
C.

#5 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Hi,

I'm using a ~4 year old laptop, 4 cores (+4 HT), 32 GB RAM,
Crucial mSATA SSD and don't notice neither the snapshotting nor the deletion of snapshots nor the transferring at all (been doing this for years now).

I'm running kernel 5.3 now, but have also been on 5.0 some time ago (but I'm on Gentoo, not Ubuntu). So I'd say this is not normal.

The first thing you'd need to check is when exactly it happens - btrbk logs the steps it is doing. Does it happen during the snapshotting, transferring, or deletion of snapshots? Anything in the kernel log?

Did you run a deduplication tool on the BTRFS volumes, or use quotas? These are the only things which come to my mind which can cause high CPU load here (but in any case, nothing should "block").

Cheers,
Oliver

Am 20.11.19 um 17:36 schrieb Christian Pernegger:

Quoted 22 lines

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

Cheers,
C.

#6 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

In reply to Christian Pernegger
Quoted 12 lines · Christian Pernegger

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

Quoted 7 lines

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

What are the mount options? And what's the workload immediate prior to the snapshot? Or does it always happen no matter the workload?

I use Btrfs on a variety of hardware and storage devices, USB flash, NVMe, hard drives, and a Samsung 940 EVO, and I can't say I experience anything like a freeze or hang. If I'm doing something like updates (dnf updates, RPM) and do a snapshot while the update is happening (bit kooky because that snapshot represents an inbetween state of the update, essentially useless except as an intentionally poking things with a stick just to see what happens) I do see a user space "hang" as a flush is required as part of the snapshot, and I see this flush using top. But so far I only see it affect the snapshot command itself (it's a delay rather than a hang). I don't see it affect GUI responsiveness.

Signature · 1 line
Chris Murphy
#7 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

In reply to Christian Pernegger
Quoted 11 lines · Christian Pernegger

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

Quoted 7 lines

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

Are you using qgroup?

With qgroup, snapshot deleting is still a problem though. (But not for snapshot creation, that shouldn't cause any slow down, unless you're using multi-level qgroups)

Thanks,
Qu

Quoted 4 lines

Cheers,
C.

#8 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Mi., 20. Nov. 2019 um 18:59 Uhr schrieb Oliver Freyermuth <[email protected]>:

Quoted 1 line

So I'd say this is not normal.

Good to hear, that means it might be fixable. The alternative would be to switch to Borg or restic, and I just don't feel comfortable with deduplication relying solely on hashes, I'm a Luddite like that.

Quoted 1 line

The first thing you'd need to check is when exactly it happens

Currently 17 minutes past the hour, which is when my cron.hourly runs, and that only runs btrbk. I can't say for certain if it happens every hour, but I'm reasonably confident.

Quoted 1 line

btrbk logs the steps it is doing. Does it happen during the snapshotting, transferring, or deletion of snapshots?

It's just configured to snapshot & prune, no transfer. A central backup server (grand name, for a white-box NAS) pulls the snapshots each night and does its own pruning. I'm not sure how to tell when exactly it happens, as I have not much agency while it is happening.

Quoted 1 line

Anything in the kernel log?

Nothing suspicious in btrbk.log, dmesg or the systemd journal. The affected things just stop reacting, then continue as if nothing had happened.

Quoted 1 line

Did you run a deduplication tool on the BTRFS volumes, or use quotas?

No to deduplication, maybe to quotas. It's possible that Timeshift enables them, how can I check?

Just had another episode:

2019-11-21T17:17:01+0100 startup v0.26.0 - - - # btrbk command line
client, version 0.26.0
2019-11-21T17:17:01+0100 snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@.20191121T171701+0100
/mnt/timeshift/backup/@ - -
2019-11-21T17:17:01+0100 snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@.20191121T171701+0100
/mnt/timeshift/backup/@ - -
2019-11-21T17:17:01+0100 snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home.20191121T171701+0100
/mnt/timeshift/backup/@home - -
2019-11-21T17:17:01+0100 snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home.20191121T171701+0100
/mnt/timeshift/backup/@home - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home-chris-.steam.20191119T161701+0100
- - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home-chris-.steam.20191119T161701+0100
- - -
2019-11-21T17:17:01+0100 finished success - - - -

I had a tail on the log, these came out in one go, no larger pauses. At first I thought, just my luck, here I am lying in wait and of course everything works, then the mini-freeze happened. CPU usage in one core spiked during the freeze, but I couldn't switch tabs from the graphs to the process list in gnome-system-monitor. Top it is, next time.

Am Mi., 20. Nov. 2019 um 19:32 Uhr schrieb Chris Murphy <[email protected]>:

Quoted 1 line

What are the mount options?

defaults, which comes out as rw,relatime,ssd,space_cache,subvolid=,subvol=, according to mount.

Quoted 1 line

And what's the workload immediate prior to the snapshot? Or does it always happen no matter the workload?

Can't guarantee "always", but ... This time I was in the process of composing this e-Mail. A couple of things open, sure, Firefox, couple of terminals, Signal, evince, deadbeat [stopped], but not doing anything much. I'd call the workload "idle", especially fs-wise. Last time I was typing at a bash prompt via gnome-terminal -- the input wouldn't show or register until it was over. It's not only i/o-intensive stuff that blocks.

Am Do., 21. Nov. 2019 um 02:51 Uhr schrieb Qu Wenruo <[email protected]>:

Quoted 1 line

Are you using qgroup?

Not knowingly. If either Timeshift or btrbk enable them, it's possible.

Cheers,
C.

#9 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am 21.11.19 um 17:44 schrieb Christian Pernegger:

Quoted 2 lines

No to deduplication, maybe to quotas. It's possible that Timeshift enables them, how can I check?

You can test with:

 $ btrfs qgroup show /
 ERROR: can't list qgroups: quotas not enabled
but none of the tools you are using should activate qgroups I think
(at least btrbk does not). 
Quoted 31 lines

Just had another episode:

2019-11-21T17:17:01+0100 startup v0.26.0 - - - # btrbk command line
client, version 0.26.0
2019-11-21T17:17:01+0100 snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@.20191121T171701+0100
/mnt/timeshift/backup/@ - -
2019-11-21T17:17:01+0100 snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@.20191121T171701+0100
/mnt/timeshift/backup/@ - -
2019-11-21T17:17:01+0100 snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home.20191121T171701+0100
/mnt/timeshift/backup/@home - -
2019-11-21T17:17:01+0100 snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home.20191121T171701+0100
/mnt/timeshift/backup/@home - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home.20191119T161701+0100 - - -
2019-11-21T17:17:01+0100 delete_snapshot starting
/mnt/timeshift/backup/btrbk-snapshots/@home-chris-.steam.20191119T161701+0100
- - -
2019-11-21T17:17:01+0100 delete_snapshot success
/mnt/timeshift/backup/btrbk-snapshots/@home-chris-.steam.20191119T161701+0100
- - -
2019-11-21T17:17:01+0100 finished success - - - -
Quoted 6 lines

I had a tail on the log, these came out in one go, no larger pauses. At first I thought, just my luck, here I am lying in wait and of course everything works, then the mini-freeze happened. CPU usage in one core spiked during the freeze, but I couldn't switch tabs from the graphs to the process list in gnome-system-monitor. Top it is, next time.

This is an interesting observation. I believe this means it is happening when the snapshot deletes are actually going to the storage, which usually happens only _after_ btrbk is finished (in case you catch it with top, a kernel thread "btrfs-cleaner" should be doing this job). Another interesting test could be to adjust btrbk configuration to:
btrfs_commit_delete = each
which will ensure the delete_snapshot operations are flushed to disk one by one, so the freeze should then correlate to the log (and might be converted from one longer freeze to multiple, contiguous smaller freezes).

Sadly, I have no idea on why this would freeze for you (well, it's the only actual I/O-heavy part when you don't do the transfers at this point in time). But maybe Qu will have a good idea.

Cheers,
Oliver

#10 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 5 lines

Am 21.11.19 um 17:44 schrieb Christian Pernegger:

Quoted 1 line

maybe to quotas. It's possible that Timeshift enables them, how can I check?

You can test with:

 $ btrfs qgroup show /

Definitely enabled, then. ... ... ... There it is: Timeshift has a pre-selected checkbox "enable BTRFS qgroups (recommended)" [translated from German].

1) How can I safely disable qgroups? Is it enough to uncheck the Timeshift option and then run btrfs quota disable or do I have to manually remove the qgroups somehow?

2) I'm wondering if this couldn't be improved. Considering qgroups are only used (in this case) for reporting on allocated space, not limiting it, and btrfs free space reporting is notoriously lazy [not meant in a bad way, can't think of a better word right now] anyway, why does anything need to block at all? Even if I were using quotas, I might prefer fuzzy quotas [that can be be hit too early/late because accounting is catching up] to a temporary standstill, as an option.

Quoted 2 lines

This is an interesting observation. I believe this means it is happening when the snapshot deletes are actually going to the storage, which usually happens only _after_ btrbk is finished (in case you catch it with top, a kernel thread "btrfs-cleaner" should be doing this job).

Ok, so btrbk runs, finishes, soon (but not immediately) after that btrfs-cleaner indeed tops the CPU charts, pegging one core to 100 %. The system is still responsive at this point. A couple of seconds into the btrfs-cleaner run, the system becomes unresponsive (top still updates throughout, though). btrfs-cleaner drops off, and btrfs-transacti[obv. cut off] takes it's place, taking 100 % CPU. Still unresponsive. As soon as btrfs-transacti is done, the system immediately recovers. Then btrfs cleaner returns, briefly, with no impact on performance. (Keep in mind that top only updates every couple seconds, it's possible btrfs-cleaner is blameless and btrfs-transacti the culprit.)

Quoted 2 lines

Another interesting test could be to adjust btrbk configuration to:
btrfs_commit_delete = each

Will do.

Cheers,
C.

#11 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 4 lines
Quoted 2 lines

Another interesting test could be to adjust btrbk configuration to:
btrfs_commit_delete = each

Will do.

Hm. No freeze, this time (with btrbk set to commit after each delete).

In other news,
- I seem to be leaking cgroups. There are currently 191 subvolumes (most of which are ro snapshots), but 547 "0/*" qgroups. Should deleting a subvolume take care of removing its (auto-created) cgroup, or does that always have to be done manually (or by setting the experimental *_qgroup_destroy options in btrbk.conf)? Any elegant ways to remove orphaned cqroups?
- Timeshift, at :00, triggers this as well, it's just less severe (maybe because that's 1 subvolume instead of 3).

Cheers,
C.

#12 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Donnerstag, 21. November 2019, 22:34:41 CET schrieb Christian Pernegger:

Quoted 17 lines
Quoted 4 lines
Quoted 2 lines

Another interesting test could be to adjust btrbk configuration to:
btrfs_commit_delete = each

Will do.

Hm. No freeze, this time (with btrbk set to commit after each delete).

In other news,
- I seem to be leaking cgroups. There are currently 191 subvolumes (most of which are ro snapshots), but 547 "0/*" qgroups. Should deleting a subvolume take care of removing its (auto-created) cgroup, or does that always have to be done manually (or by setting the experimental *_qgroup_destroy options in btrbk.conf)? Any elegant ways to remove orphaned cqroups?
- Timeshift, at :00, triggers this as well, it's just less severe (maybe because that's 1 subvolume instead of 3).

Quoted 2 lines

Cheers,
C.

As Qu said, the freezes should only happen on snapshot deletion. Depending on how you have btrbk configured and how regularly it runs, not every btrbk run will delete snapshots. Therefor not every run will cause the system to lock up.

On a side note, I am also really annoyed by the lockups caused by qgroups. On my Gentoo systems (which use btrbk) I have it disabled for that reason, but I left it on on my openSUSE laptop (a Dell XPS 13 9360), which locks up for about 15-30 minutes while cleaning up snapshots a few times a week (usually after reboots or after "zypper dup"). Of course, that's with snapshots active for /home, which I do so that the file system doesn't change out from under borg while it's running. I'm tentatively considering turning it off there, too, but I'll experiment with the snapper configuration first.

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup

#13 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 5 lines · Marc Joliet

On a side note, I am also really annoyed by the lockups caused by qgroups. On my Gentoo systems (which use btrbk) I have it disabled for that reason, but I left it on on my openSUSE laptop (a Dell XPS 13 9360), which locks up for about 15-30 minutes while cleaning up snapshots a few times a week (usually after reboots or after "zypper dup").

15 seconds is not at all acceptable on a desktop system, 15 minutes is atrocious. A computer that appears to hang for 15 seconds, it is completely reasonable for ordinary users to consider has totally faceplanted, will not recover, and to force power off. The distribution really needs to do something about that kind of negative user experience.

And by the way, I've recently done some unprivileged compilations of webkitgtk, with default options that cause n cores +2 to be used, eating all available RAM and swap, and quickly totally hanging the system while swap thrashing and basically acting like a fork bomb. I'm using Btrfs for the rootfs as well as user home for this compile, and have done hundreds of forced power offs during these events and have seen exactly zero corruptions or Btrfs complaints. So at least there's that, however unscientific a sample that is.

Signature · 1 line
Chris Murphy
#14 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Freitag, 22. November 2019, 02:36:56 CET schrieb Chris Murphy:

Quoted 7 lines · Marc Joliet
Quoted 5 lines

On a side note, I am also really annoyed by the lockups caused by qgroups.
On my Gentoo systems (which use btrbk) I have it disabled for that
reason, but I left it on on my openSUSE laptop (a Dell XPS 13 9360), which locks up for about 15-30 minutes while cleaning up snapshots a few times a week (usually after reboots or after "zypper dup").

Quoted 6 lines

15 seconds is not at all acceptable on a desktop system, 15 minutes is atrocious. A computer that appears to hang for 15 seconds, it is completely reasonable for ordinary users to consider has totally faceplanted, will not recover, and to force power off. The distribution really needs to do something about that kind of negative user experience.

Sadly, I can't say if it's better without snapshotting /home, because I hadn't accumulated many / snapshots at that point in time. It might have gotten worse even with only / being snapshotted. But like I said, I'll experiment with configuring snapper before blaming SUSE. I believe the installation even recommends against snapshotting /home, but hey, I wanted to do it anyway :-) .

But to be precise, it's not locked up continuously during snapshot deletion. Occasionally I'll be able to operate my desktop for a few seconds, and if I leave top running in a GUI terminal (in my case konsole), I'll see it updating (almost) the entire time. My guess (emphasis on *guess*) is that the qgroups update is holding some lock that is preventing other I/O from finishing, thus locking up any application that wants to write to disk and isn't doing so concurrently (maybe Plasma is blocking on fsync() at the time?).

Quoted 8 lines

And by the way, I've recently done some unprivileged compilations of webkitgtk, with default options that cause n cores +2 to be used, eating all available RAM and swap, and quickly totally hanging the system while swap thrashing and basically acting like a fork bomb. I'm using Btrfs for the rootfs as well as user home for this compile, and have done hundreds of forced power offs during these events and have seen exactly zero corruptions or Btrfs complaints. So at least there's that, however unscientific a sample that is.

My experience has also been that forced reboots don't cause any damage, even though I usually only have to do them rarely [0]. I mean, with COW it should be expected to be safe.

[0] I have two main situations where this happens: The first are RCU stalls that cause my desktop to get hung up (happens during bootup occasionally, shortly between the boot loader and the login screen), but also recently started affecting my home server. The second only affects my home server (a used small business server), namely a wonky e1000e NIC, which I only recently learned are sometimes buggy are known for causing servers to crash. The workaround is apparently to turn off TSO and GSO, and sometimes also GRO, but I've been able to get away with only the first two without experiencing any more crashes thus far. Interestingly enough the RCU stalls happened shortly after I did that.

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup

#15 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Samstag, 23. November 2019, 00:21:18 CET schrieben Sie:

Quoted 23 lines

Am Freitag, 22. November 2019, 02:36:56 CET schrieb Chris Murphy:

Quoted 8 lines · Marc Joliet
Quoted 6 lines

On a side note, I am also really annoyed by the lockups caused by qgroups.
On my Gentoo systems (which use btrbk) I have it disabled for that reason, but I left it on on my openSUSE laptop (a Dell XPS 13 9360), which locks up for about 15-30 minutes while cleaning up snapshots a few times a week (usually after reboots or after "zypper dup").

Quoted 6 lines

15 seconds is not at all acceptable on a desktop system, 15 minutes is atrocious. A computer that appears to hang for 15 seconds, it is completely reasonable for ordinary users to consider has totally faceplanted, will not recover, and to force power off. The distribution really needs to do something about that kind of negative user experience.

Sadly, I can't say if it's better without snapshotting /home, because I hadn't accumulated many / snapshots at that point in time. It might have gotten worse even with only / being snapshotted. But like I said, I'll experiment with configuring snapper before blaming SUSE. I believe the installation even recommends against snapshotting /home, but hey, I wanted to do it anyway :-) .

Quoted 8 lines

But to be precise, it's not locked up continuously during snapshot deletion. Occasionally I'll be able to operate my desktop for a few seconds, and if I leave top running in a GUI terminal (in my case konsole), I'll see it updating (almost) the entire time. My guess (emphasis on *guess*) is that the qgroups update is holding some lock that is preventing other I/O from finishing, thus locking up any application that wants to write to disk and isn't doing so concurrently (maybe Plasma is blocking on fsync() at the time?).

So just to follow up on this, reducing the total number of snapshots and increasing the time between their creation from hourly to once every six hours did help a *little* bit. However, about a week ago I decided to try an experiment and added the "autodefrag" mount option (which I don't usually do on SSDs), and that helped *massively*. Ever since, snapper-cleanup.service runs without me noticing at all!

[ What made me try it was that booting the laptop and logging in started getting really slow and top was showing several btrfs-endio threads hogging the CPU, *before* snapper-cleanup.service or anything else specific to btrfs was running (their activity usually coincided with KDE Baloo activity), i.e., general I/O was performing badly. ]

Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup

#16 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

In reply to Christian Pernegger

Am 21.11.19 um 21:30 schrieb Christian Pernegger:

Quoted 3 lines

Definitely enabled, then. ... ... ... There it is: Timeshift has a pre-selected checkbox "enable BTRFS qgroups (recommended)" [translated from German].

Since I've never used qgroups myself, I'll only comment on the parts where I can. However, I would say "(recommended)" just to get an estimate of space consumption is a rather hard label for the option in Timeshift.

You can check the known issues on qgroups:
https://btrfs.wiki.kernel.org/index.php/Quota_support#Known_issues
This contains, amongst other things, the observed performance issues and also:
"- After deleting a subvolume, you must manually delete the associated qgroup." which you observe, too. But it does indeed seem btrbk can help out here:
https://github.com/digint/btrbk/issues/49

Manpages of btrfs-quota and btrfs-qgroup contain quite some warnings about the existence
of these known issues, the status page at:
https://btrfs.wiki.kernel.org/index.php/Status
links them, etc. So I believe the recommendation by Timeshift is somewhat hefty.
Other downstreams (see e.g. https://wiki.debian.org/Btrfs or https://wiki.archlinux.org/index.php/Btrfs#Quota ) explicitly recommend not to use qgroup unless really needed.

Apparently this has also been raised to the developer:
https://github.com/teejee2008/timeshift/issues/127
which has at least led to the addition of the checkmark to allow not enabling qgroup.

Quoted 7 lines

2) I'm wondering if this couldn't be improved. Considering qgroups are only used (in this case) for reporting on allocated space, not limiting it, and btrfs free space reporting is notoriously lazy [not meant in a bad way, can't think of a better word right now] anyway, why does anything need to block at all? Even if I were using quotas, I might prefer fuzzy quotas [that can be be hit too early/late because accounting is catching up] to a temporary standstill, as an option.

You can check e.g. the man page btrfs-quota(8) for a short discussion on why doing quota correctly with btrfs is not as easy as it may seem.
I'll leave more comments (and how to disable them safely) to those who have experience with qgroups ;-).

Quoted 4 lines
Quoted 2 lines

Another interesting test could be to adjust btrbk configuration to:
btrfs_commit_delete = each

Will do.

...

Quoted 1 line

Hm. No freeze, this time (with btrbk set to commit after each delete).

That might be a red herring if there was just less to delete, as Marc Joliet pointed out, at least, I think this means we identified the reason for the freezes you get.

Cheers,
Oliver

#17 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Fr., 22. Nov. 2019 um 00:57 Uhr schrieb Oliver Freyermuth <[email protected]>:

Quoted 4 lines
Quoted 1 line

2) I'm wondering if this couldn't be improved. [...]

You can check e.g. the man page btrfs-quota(8) for a short discussion on why doing quota correctly with btrfs is not as easy as it may seem.

I've read that and I appreciate the difficulties in getting accurate usage information (or even defining what that means) from a COW filesystem. IMHO, performance, and the trade-off between performance and up-to-the-minute accuracy are separate issues.

FWIW, running btrfs quota disable, enable, and rescan got rid of the orphan qgroups. The full rescan ran for all of 3 seconds and didn't block.

Cheers,
C.

#18 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 7 lines · Christian Pernegger

Am Fr., 22. Nov. 2019 um 00:57 Uhr schrieb Oliver Freyermuth <[email protected]>:

Quoted 4 lines
Quoted 1 line

2) I'm wondering if this couldn't be improved. [...]

You can check e.g. the man page btrfs-quota(8) for a short discussion on why doing quota correctly with btrfs is not as easy as it may seem.

Quoted 8 lines

I've read that and I appreciate the difficulties in getting accurate usage information (or even defining what that means) from a COW filesystem. IMHO, performance, and the trade-off between performance and up-to-the-minute accuracy are separate issues.

FWIW, running btrfs quota disable, enable, and rescan got rid of the orphan qgroups. The full rescan ran for all of 3 seconds and didn't block.

BTW, for the empty qgroup auto delete, we have pending patch for that already.
Just not merged yet.

https://patchwork.kernel.org/patch/11195067/

But still, for snapshot deletion part, there is still a performance impact. (For completely independent subvolume, IIRC there is a quick path for it, thus no performance penalty then)

Thanks,
Qu

Quoted 4 lines

Cheers,
C.

#19 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am Fr., 22. Nov. 2019 um 13:34 Uhr schrieb Qu Wenruo <[email protected]>:

Quoted 1 line

But still, for snapshot deletion part, there is still a performance impact.

Ok. It's just that I'd have expected *slower* write and read performance until everything's settled, maybe sync writes taking noticeably longer than usual, not that all user input blocks across the whole system regardless of fs activity.

Cheers,
C.

#20 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 7 lines · Christian Pernegger

Am Fr., 22. Nov. 2019 um 13:34 Uhr schrieb Qu Wenruo <[email protected]>:

Quoted 1 line

But still, for snapshot deletion part, there is still a performance impact.

Ok. It's just that I'd have expected *slower* write and read performance until everything's settled, maybe sync writes taking noticeably longer than usual, not that all user input blocks across the whole system regardless of fs activity.

The slowdown happens in commit transaction, and with commit transaction, a lot of operation is blocked until current transaction is committed.

That's why it blocks everything.

We had tried our best to reduce the impact, but deletion is still a big problem, as it can cause tons of extents to change their owner, thus cause the problem.

In short, unless you really need to know how many bytes each snapshots really takes, then disable qgroup.

And BTW, for "many" subvolumes/snapshots, I guess we mean 20. 200 is already prone to cause problem, not only qgroups, but also send.

So it's also recommended to reduce the number of snapshots.

Thanks,
Qu

Quoted 4 lines

Cheers,
C.

#21 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Am So., 24. Nov. 2019 um 01:38 Uhr schrieb Qu Wenruo <[email protected]>:

Quoted 7 lines

In short, unless you really need to know how many bytes each snapshots really takes, then disable qgroup.

And BTW, for "many" subvolumes/snapshots, I guess we mean 20. 200 is already prone to cause problem, not only qgroups, but also send.

So it's also recommended to reduce the number of snapshots.

I've disabled qgroups for now, we'll see how that goes. These are personal desktops, they would have been nice to have, that's all. Sadly that means that they probably won't work on any storage setup complex enough for them to be really useful, either, yet. If btrfs scales so badly with the number of subvolumes that having >20 at a time should be avoided, doesn't that kill a lot of interesting use-cases? My "time machine" desktop setup, certainly, but anything with a couple of users or VMs would chew through that 20 pretty quickly, even before snapshots. Which leaves the LVM use-case (snapshot, backup the snapshot, delete the snapshot).

Quoted 8 lines

The slowdown happens in commit transaction, and with commit transaction, a lot of operation is blocked until current transaction is committed.

That's why it blocks everything.

We had tried our best to reduce the impact, but deletion is still a big problem, as it can cause tons of extents to change their owner, thus cause the problem.

Sure, but why does it *have to* block? Couldn't the intent to delete the subvolume be committed, the metadata changes / actual deletion happen at leisure? Yes, if qgroups are on, then the qgroup info will be behind, but so what? At least I think that lax/lazy qgroups would be a nice option to have.
Also, I still don't get why disabling qgroups, reenabling them and doing a full rescan is lightning fast (and non-blocking), while just leaving them on results in the observed behaviour.

Cheers,
C.

#22 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 9 lines · Christian Pernegger

Am So., 24. Nov. 2019 um 01:38 Uhr schrieb Qu Wenruo <[email protected]>:

Quoted 7 lines

In short, unless you really need to know how many bytes each snapshots really takes, then disable qgroup.

And BTW, for "many" subvolumes/snapshots, I guess we mean 20. 200 is already prone to cause problem, not only qgroups, but also send.

So it's also recommended to reduce the number of snapshots.

Quoted 10 lines

I've disabled qgroups for now, we'll see how that goes. These are personal desktops, they would have been nice to have, that's all. Sadly that means that they probably won't work on any storage setup complex enough for them to be really useful, either, yet. If btrfs scales so badly with the number of subvolumes that having >20 at a time should be avoided, doesn't that kill a lot of interesting use-cases? My "time machine" desktop setup, certainly, but anything with a couple of users or VMs would chew through that 20 pretty quickly, even before snapshots. Which leaves the LVM use-case (snapshot, backup the snapshot, delete the snapshot).

BTW, that 20 number means 20 snapshots (they all have some shared tree blocks).

If it's 20 subvolume (no shared tree/data between each), then it counts as 1.

The main time consuming part is the shared tree/data check, as btrfs uses indirect way to record them on-disk, forcing us to do complex walk-back.

Thankfully, we have some plan to improve it.

Quoted 6 lines
Quoted 6 lines

The slowdown happens in commit transaction, and with commit transaction, a lot of operation is blocked until current transaction is committed.

That's why it blocks everything.

Quoted 7 lines
Quoted 3 lines

We had tried our best to reduce the impact, but deletion is still a big problem, as it can cause tons of extents to change their owner, thus cause the problem.

Sure, but why does it *have to* block? Couldn't the intent to delete the subvolume be committed, the metadata changes / actual deletion happen at leisure?

Unfortunately, not that easy.
We have already delayed a lot of metadata operation, and commit transaction is the only time we get a consistent metadata view.

That's why it has to happen at that critical section.

Quoted 2 lines

Yes, if qgroups are on, then the qgroup info will
be behind, but so what?

It's already behind.

Quoted 2 lines

At least I think that lax/lazy qgroups would
be a nice option to have.

Qgroup is bond to delayed extent tree updates.
While extent tree update is already delayed to transaction commit time, if it's further delayed, the consistency of the fs will be corrupted.

The plan to solve it is to introduce a global cache for backref walk, which would not only benefit qgroup, but also send with reflink.

Although there will be some new challenges, we will see if the cache will be worthy.

Thanks,
Qu

Quoted 7 lines

Also, I still don't get why disabling qgroups, reenabling them and doing a full rescan is lightning fast (and non-blocking), while just leaving them on results in the observed behaviour.

Cheers,
C.

#23 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

In reply to Christian Pernegger
Quoted 10 lines · Christian Pernegger

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

Snapshot delete is pretty aggressive with IO and can force a lot of commits if you are modifying a lot of metadata pages between snapshots. Generally I get a coffee when my 1TB NVME systems decide it's time to drop a snapshot, as the system can effectively hang for a few minutes while btrfs-cleaner runs. On performance-critical systems we only ever have one snapshot active on the filesystem at a time, and we only create it once a day for backups. I'd love a way to throttle btrfs-cleaner so it's not so aggressive with IO and CPU.

Snapshot create has unbounded running time on 5.0 kernels. The creation process has to flush dirty buffers to the filesystem to get a clean snapshot state. Any process that is writing data while the flush is running gets its data included in the snapshot flush, so in the worst possible case, the snapshot flush never ends (unless you run out of disk space, or whatever was writing new data stops, whichever comes first).

Anything that needs to take a sb_writer lock (which is almost everything that modifies the filesystem) will hang until the snapshot create is done; however, processes that are reading the filesystem will not be obstructed. This can lead to starvation of the writing processes. cgroups and ionice won't help here--the block layer doesn't detect waits for sb_writers (there is no associated block device for those, so they're invisible to the block layer), so it doesn't know that writer processes are waiting for IO, and all the writers' IO bandwidth gets reallocated to the reader processes, making for long-lasting priority inversions. The IO pressure stall subsystem reads _zero_ IO pressure even though writing processes are continuously blocked for hours.

On small systems, this is all over in a second or less. On bigger fileservers, I've had single snapshot creates run for many hours. As a workaround, I have some scripts that freeze processes that write to the disk while 'btrfs sub create' runs, to force the snapshot create to finish in a timely manner. I think I saw some patches going into later 5.x kernels that solve the problem in the kernel, too (writes that occur after the snapshot creation starts are not included in the snapshot any more).

Quoted 10 lines

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

Cheers,
C.

#24 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 40 lines · Zygo Blaxell
Quoted 10 lines · Christian Pernegger

Hello,

I've decided to go with a snapshot-based backup solution for our new Linux desktops -- thank you for the timely thread --, namely btrbk. A couple of subvolumes for different stuff, with hourly snapshots that regularly go to another machine. Brilliant in theory, less so in practice, because every time btrbk runs, the box'll freeze for a few seconds, as in, Firefox and LibreOffice, for instance, become entirely unresponsive, games hang and so on. (AFAICT, all it does is snapshot each subvolume and delete ones that are out of the retention period.)

Snapshot delete is pretty aggressive with IO and can force a lot of commits if you are modifying a lot of metadata pages between snapshots. Generally I get a coffee when my 1TB NVME systems decide it's time to drop a snapshot, as the system can effectively hang for a few minutes while btrfs-cleaner runs. On performance-critical systems we only ever have one snapshot active on the filesystem at a time, and we only create it once a day for backups. I'd love a way to throttle btrfs-cleaner so it's not so aggressive with IO and CPU.

Snapshot create has unbounded running time on 5.0 kernels. The creation process has to flush dirty buffers to the filesystem to get a clean snapshot state. Any process that is writing data while the flush is running gets its data included in the snapshot flush, so in the worst possible case, the snapshot flush never ends (unless you run out of disk space, or whatever was writing new data stops, whichever comes first).

Anything that needs to take a sb_writer lock (which is almost everything that modifies the filesystem) will hang until the snapshot create is done; however, processes that are reading the filesystem will not be obstructed. This can lead to starvation of the writing processes. cgroups and ionice won't help here--the block layer doesn't detect waits for sb_writers (there is no associated block device for those, so they're invisible to the block layer), so it doesn't know that writer processes are waiting for IO, and all the writers' IO bandwidth gets reallocated to the reader processes, making for long-lasting priority inversions. The IO pressure stall subsystem reads _zero_ IO pressure even though writing processes are continuously blocked for hours.

Quoted 7 lines

On small systems, this is all over in a second or less. On bigger fileservers, I've had single snapshot creates run for many hours. As a workaround, I have some scripts that freeze processes that write to the disk while 'btrfs sub create' runs, to force the snapshot create to finish in a timely manner. I think I saw some patches going into later 5.x kernels that solve the problem in the kernel, too (writes that occur after the snapshot creation starts are not included in the snapshot any more).

Nope, the patch I'm thinking of is dated Nov 1 *2018* and is already in 5.0. So either that fix is ineffective, or the slow snapshots are caused by something else.

Quoted 10 lines
Quoted 10 lines

I'm aware that having many snapshots can impact performance of some operations, but I didn't think that "many" <= 200, "impact" = stop dead and "some operations" = light desktop use. These are decently specced, after all (Zen 2 8/12 core, 32 GB RAM, Samsung 970 Evo Plus). What I'm asking is, is this to be expected, does it just need tuning, is the hardware buggy, the kernel version (Ubuntu 18.04.3 HWE, their 5.0 series) a stinker, something else awry ...?

Cheers,
C.

#25 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

In reply to Zygo Blaxell

Am Do., 21. Nov. 2019 um 23:22 Uhr schrieb Zygo Blaxell <[email protected]>:

Quoted 2 lines

Snapshot delete is pretty aggressive with IO [...] can effectively hang for a few minutes while btrfs-cleaner runs.

It's doesn't look like it's btrfs-cleaner that blocks here, though, more like it's btrfs-transacti.

Quoted 1 line

Snapshot create has unbounded running time on 5.0 kernels.

It looks to me like delete, not create, is the culprit here.

Quoted 2 lines

Anything that needs to take a sb_writer lock (which is almost everything that modifies the filesystem) will hang until the snapshot create is done;

It's not just fs activity, either. Even if I'm just typing in LibreOffice or at a bash prompt, the input isn't registered during the freeze (it's buffered, so it comes out all at once in the end).

Cheers,
C.

#26 lore raw

Re: freezes during snapshot creation/deletion -- to be expected? (Was: Re: btrfs based backup?)

Quoted 7 lines · Christian Pernegger

Am Do., 21. Nov. 2019 um 23:22 Uhr schrieb Zygo Blaxell <[email protected]>:

Quoted 2 lines

Snapshot delete is pretty aggressive with IO [...] can effectively hang for a few minutes while btrfs-cleaner runs.

It's doesn't look like it's btrfs-cleaner that blocks here, though, more like it's btrfs-transacti.

It's hard to tell. btrfs-transaction does a lot of work for other threads. If you have kernel stacks enabled,

watch -n.1 cat /proc/<pid of btrfs-cleaner>/stack

will show you what btrfs-cleaner is up to. If it's something like 'wait_for_commit' then btrfs-cleaner dumped a bunch of work on btrfs-transaction, and now btrfs-transaction is trying to catch up.

Quoted 10 lines
Quoted 1 line

Snapshot create has unbounded running time on 5.0 kernels.

It looks to me like delete, not create, is the culprit here.

Quoted 2 lines

Anything that needs to take a sb_writer lock (which is almost everything that modifies the filesystem) will hang until the snapshot create is done;

It's not just fs activity, either. Even if I'm just typing in LibreOffice or at a bash prompt, the input isn't registered during the freeze (it's buffered, so it comes out all at once in the end).

IO pressure, especially blocked writes, can delay memory allocations on Linux. That stops almost everything dead in a modern GUI.

If you can log into the box from another machine you might be able to watch what it's doing with 'top' etc.

On the other hand, from the other messages in this thread, it sounds like you're using qgroups, which multiplies everything I said above by 1000. qgroups is all in-kernel CPU, too, so userspace can't preempt it.

Quoted 2 lines

Cheers,
C.

#27 lore raw
In reply to Ulli Horlacher

Hi,

I'm not sure if the btrfs list is the correct place for a generic answer - but I'll try to give one mentioning all the backup solutions I have collected experience with (all open source, of course).

1) btrbk ( https://github.com/digint/btrbk )
I use it on all my personal machines, both for local snapshotting (to unroll my own mistakes easily...) and for sending the incrementals
to an external storage. It's basically a well-working btrfs send / receive automation, so it needs btrfs at both ends (or becomes less efficient), which may not match your use case.

2) Borg Backup ( https://borgbackup.readthedocs.io/en/stable/ )
I use this whenever I do not have btrfs at one / both ends. It can also do encryption, compression and deduplication, purge old incrementals without ever doing a full backup, you can even mount your backups.
I use this for some smaller machines (e.g. on a Raspberry Pi) and we use it on our infrastructure for some configuration backups.

3) Restic ( https://restic.readthedocs.io/en/latest/ )
Restic is (feature-wise) like borg (but no compression yet). The main difference is that it can (but does not have to) back up to cloud-like storages such as S3. We intend to use this heavily to a local Ceph storage system with 3x replication offering S3/Swift via Rados Gateway nodes.
If you want something less heavy than a Ceph cluster (we love it, it does not bite!) you can try minio ( https://min.io/ ). I never used minio myself, but only heard good things about it.

4) Duplicati ( https://www.duplicati.com/ )
Like Borg / Restic (can also talk S3 if wanted, or store to a file system, also does compression).
The main advantage here is that it has a GUI. Probably not interesting for your use case, but we intend to recommend that to our users with Windows / MacOS X who may prefer some buttons to click.

5) Since you mention VMs in your signature, I'll also mention:
https://benji-backup.me/
http://backy2.com/
https://bitbucket.org/flyingcircus/backy
I'll personally recommend benji here, due to a large featureset, very active development and high efficiency. It does differential backups of RBD volumes, so it will only be really useful to you if you use Ceph RBD (you can also get it to work with LVM and raw block devices, I think).
You can find some of our experiences with it here:
https://indico.cern.ch/event/765214/contributions/3517132/

I think all of these are not too complex (of course, they only work well if your infrastructure matches them)
since you can essentially arrive at a working backup and restore in a few minutes.
I'll also add that for almost all of our servers, we do not do any backups at all - file servers and services with data have their storage replicated to their HA partner node(s), and all configuration is "backed up" by having it completely in Foreman / Puppet, so a machine can be reinstalled at the push of a button.

The main functionalities you give up with your original idea (rsync to one replicated node) is that you do not have deduplication built-in and would need to do encryption at the source yourself if needed. Also, you would have to do regular "full" backups and think about how you can keep incrementals - rsnapshot (which nowadays seems rather dead) could do something similar via rsync with hardlinks, but that meant you had tons of files which no FS really likes, and would always have to store a full file if a single byte changed.

Cheers and hope that helps,
Oliver

Am 12.11.19 um 19:34 schrieb Ulli Horlacher:

Quoted 25 lines

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

To have a secondary backup I will use btrfs send / receive,

Any comments on this? Or better suggestions?

The backup software must be open source!

#28 lore raw
In reply to Ulli Horlacher

12.11.2019 20:34, Ulli Horlacher пишет:

Quoted 11 lines

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Quoted 7 lines

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

To have a secondary backup I will use btrfs send / receive,

Check my message with subject *"**Read-only snapshot send speed very slow after modify original data. Need help <https://www.spinics.net/lists/linux-btrfs/msg94128.html>*/"./

/Very-very slow send read-only snapshot after modify original rw subvol if compress. In some cases ~8-15 hour per snapshot./

/100cpu load and send only 5-100mb diff./
/There is no reaction
/

Quoted 5 lines

Any comments on this? Or better suggestions?

The backup software must be open source!

Signature · 2 lines
С уважением, Михаил
067-786-11-75
#29 lore raw
In reply to Ulli Horlacher
Quoted 18 lines · Ulli Horlacher

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

All backup software I know is either too primitive (e.g. no versioning) or very complex and needs a long time to master it.

My new idea is:

Set up a backup server with btrfs storage (with compress mount option), the clients do their backup with rsync over nfs.

For versioning I make btrfs snapshots.

Quoted 7 lines

To have a secondary backup I will use btrfs send / receive,

Any comments on this? Or better suggestions?

The backup software must be open source!

Borg [1] backup on the clients. That will get you:

* Automatic 'versioning' without needing snapshots.
* Automatic compression and deduplication of the backups (without needing BTRFS to do either).
* Automatic encryption (if you want it).
* The ability to mount your backups like a filesystem (through FUSE).
* All in a layout that's reasonably friendly to copy between systems with tools like rsync or rclone.

Borg's big thing is that it does reference-counted deduplication of the individual blocks of the backup, so incrementals take up next to no space or time but still give you a full view of the backed up filesystem with each snapshot. It also has support for accessing a backup server over SSH, which is a bit more efficient than using something like NFS.

For the copy to the secondary backup, you could just use rsync to mirror the backups done with Borg (or alternatively, us rclone to mirror them offsite to cloud storage).

[1] https://borgbackup.readthedocs.io/en/stable/

#30 lore raw
In reply to Ulli Horlacher
Quoted 5 lines · Ulli Horlacher

I need a new backup system for some servers. Destination is a RAID, not tapes.

So far I have used a self written shell script. 25 years old, over 1000 lines of (HORRIBLE) code, no longer maintenable :-}

Thanks for all your suggestions, but I found myself a really easy solution just with btrfs+rsync.

On the clients I use:

root@tandem:~# grep backup /etc/fstab 
mutter:/backup/rsync/tandem     /backup         nfs ro,tcp,soft,retrans=1 0 0
root@tandem:~# cat bin/rsync_backup 
#!/bin/bash

exclude='
--exclude=.snapshot
--exclude=.del
--exclude=*.iso
--exclude=tmp/*
--exclude=backup
'

if [ ! -t 0 ]; then
exec >>/var/log/backup.log
chmod 600 /var/log/backup.log
echo
date +"%Y-%m-%d %H:%M:%S"
fi

mount /backup
mount -o remount,rw /backup || exit 1
rsync -vaxH --delete $exclude / /export /backup/
touch /backup/.ready
mount -o remount,ro /backup

On the backup server I use:

root@mutter:/backup/rsync# grep backup /etc/crontab 
0  *    * * *   root    /backup/rsync/snapshot >/dev/null
root@mutter:/backup/rsync# cat snapshot 
#!/bin/bash

PATH="$PATH:/opt/btrfs-tools/bin"

for i in /backup/rsync/*/.ready; do
if [ -f "$i" ]; then

    rm $i
    snaprotate rsync 10 $(dirname $i)

fi
done

That's all! Works like a charm :-)
And substitutes an unmaintainable 1000+ lines shell script.

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>