btrfs auto snapshot

sorry forgot to follow the protocol, now included RFC in the subject.

Hi,

Appears that no one is working on the auto-snapshot feature for btrfs, so here I am implementing the same.

Below is a draft on the feature list. Any comments / questions / suggestions are welcome, please do let me know.

btrfs auto snapshot feature will include:
Initially:
- configurable timely snapshots
- uses services and crontab to schedule
- Gnome integration
- snapshot rollback and cleanups
- snapshot trashing based on available space
- snapshot destination will be subvol/.btrfs/snapshot@<time> and snapshot/.btrfs/snapshot@<time> for subvolume and snapshot respectively

Later:
-integration with (a ?) backup software
- snapshot trashing per backup confirmation
- rollback from the backup server

Challenges:
- rollback per file or dir instead of entire snapshot-rollback ?
- integrating into the btrfs gui ?
- some FS (samfs) support continues backup
- do we need a new role for the snapshot management ? zfs does.
- we don't need a snapshot if the master file-system didn't change at all
- snapshots are writable by default, hope that sudo-er doesn't modify the snapshot - do we need to implement a kind of read-only snapshot ?

Thanks, -Anand

And a rough implementation design is here below. (As of now this does not include the GNOME integration since I have no idea how to do that).

Further, implementation will contain 2 new files /etc/init.d/btrfs and /<path>/btrfs-auto-snapshot,

any idea where does a file like btrfs-auto-snapshot should be placed (with its purpose as mentioned below).

-------
1.
File: /etc/init.d/btrfs :
This will contain the global start and stop knob (something like service btrfs start)

2.
Crontab: Upon start, crontab will be updated with something like the following.
every 15min <path>/btrfs-auto-snapshot cleanup>
every 15min /<path>/btrfs-auto-snapshot 15min
every hr /<path>/btrfs-auto-snapshot hr
every day /<path>/btrfs-auto-snapshot day
every month /<path>/btrfs-auto-snapshot month
every year /<path>/btrfs-auto-snapshot year

3.
File: /<path>/btrfs-auto-snapshot :
to process the call from cronjob.
- check the config and check the target fs for the snapshot
- check the space in the target FS
- call btrfs cli to take snapshot (provide src and destination)
- check if snapshot cleanup is required

to process the calls from '/etc/init.d/btrfs'

configuration setting.
- to set which btrfs fs will participate in the auto-snapshot
- to indicate if auto snapshot has to stop when target FS is 80% full

---------

Thanks, Anand

[...]

I think that you need to be careful not to impose your idea of when to take snapshots and how long to keep them onto the design. For example why take snapshots every 15 minutes? Why not every 10 or every hour?

Why treat monthly snapshots as special when it does not fit into most working weeks? would weekly be more logical? What about 2 weekly (When I worked at Nokia, internal releases where done on Tuesday of each even numbered week, so we would have wanted the snapshot taken on that day to be retained longer than snapshots taken on other days, or Tuesdays in odd numbered weeks.)

I think a more flexible design would be to allow the user to specify (via a config file for each subvolume) a label for each type of snapshot and how long to keep snapshots depending on when they are taken. This can be done using syntax similar to crontab:

Example

# Format: <label> <crontab like fields> <retention time>

# Keep 15 min snapshots for 6 hours.
15_minute */15 * * * * 6 hour

# Keep hourly snapshots for 5 day
hourly 0 * * * * 5 day

# Keep daily snapshots for 21 days. Use the noon snapshot
daily 0 12 * * * 21 day

# Keep weekly snapshots for 6 months, Use the Tuesday snapshot # TODO: we would need a week number field to specify even
# numbered weeks only.
weekly 0 12 * * 2 6 mon

The btrfs-auto-snapshot script would need to make sure that the crontab entry that takes snapshots runs frequently enough.

David Pottage

Error compiling committee.c To many arguments to function.

David,

crontab is anyways changeable by the admin, I think we can have that flexibility.

agreed. weekly is more important. (I had that in mind but missed it when writing, sorry for that).

simple and nice idea. thanks for explaining, will try to get this in the initial release.

Cheers, Anand

Hi Anand,

On Wed, Aug 17, 2011 at 11:24, Anand Jain <[email protected]> wrote= :

).

Very cool idea! With regards to the Gnome integration, you might want t=
o take
a look at what the Solaris folks did for Nautilus - I really liked the UI and the
integration with ZFS:

http://src.opensolaris.org/source/xref/jds/spec-files/branches/gnome-2-=
30/patches/nautilus-14-zfs-snapshot.diff

Bye,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 LenZ
--
=C2=A0Lenz Grimmer <[email protected]> -=C2=A0 http://www.lenzg.net/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Anand and all,

On 2011-08-17 T 10:15 +0800 Anand Jain wrote:
=20

thanks for bringing this up! The group of features you are listing is indeed of high interest for people using btrfs.

That said, not only have other people though about this, but a lot of your question already have been implemented in "snapper", and open source infrastructure developed as part of openSUSE and SUSE Linux Enterprise.

Please see:

	http://en.opensuse.org/Portal:Snapper
	http://en.opensuse.org/openSUSE:Snapper_install
	http://lizards.opensuse.org/2011/04/01/introducing-snapper/

Source code is here:
http://gitorious.org/opensuse/snapper

"snapper" will be part of openSUSE 12.1 and SUSE Linux Enterprise 11 Service Pack 2, and is available as part of the respective Beta releases and Milestones already.

snapper's concept in short:
- shared library to make the functionality available to other tools as well
- libsnapper is implemented on top of the btrfsprogs
- cmdline tool "snapper"
- global configuration file
/etc/sysconfig/snapper
- one configuration file per subvolume to be snapshotted /etc/snapper/configs/<config-name>
I call this a "single configuration" going forward. Here also policies for time based snapshotting and cleanup are to be configured.
- Integration into SUSE's management framework (YaST2/zypper), however, "snapper" should work independent of those, i.e. usable on other distributions easily.

Let me go through the single features quickly and list the matching snapper functionality.
=20

Yes. Configured per single configuration

Yes.

I more see a need for integration into systems management frameworks.

Yes. Rules for cleanups (time based, number of snapshots) per single configuration.

// not yet done.

snapshot destination is "/.snapshots/<number>/",=20

Timestamp and Description of a snapshot are stored in a small XML file /.snapshots/<number>/info.xml". One small file per snapshot.

[...]

snapper implements "rollback" on a FILE level only.

To differentiate this way of "rolling back" from jumping into another snapshot, we call it
"undochange"
for now. This keeps the option to also manage a full per snapshot-rollback in a later point int time.

[...]

y

snapper treats snapshots as read only snapshots, i.e. when doing a rollback - aehem, I should say "undochange" - only the "master" volume will be changed, not the single snapshots. We are aware that this has pros and cons. But that's another discussion.

I hope that this is a starting point for you.

Enjoy "snapper".

so long -
MgE

--=20
Matthias G. Eckermann Senior Product Manager SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH Maxfeldstra=C3=9Fe 5 90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3= =BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks MgE. snapper is cool, does most the stuff required here.

however the challenging part will be to keep the number of tools (to manage btrfs) at a limit 1 or 2 max. (too many tools to manage btrfs is most likely to confuse).

Cheers, Anand

I've already done something similar. I take hourly, daily, weekly, and monthly snapshots of my /home subvolume. Here's the script I've created for this:

#! /bin/bash

if [ "$#" -ne 2 ]; then
echo Usage $0 SNAPSHOT_PREFIX NUM_SNAPSHOTS
exit 1
fi

SNAPS=/var/lib/btrfs-root/__snapshot/home

btrfs subvolume snapshot -r /home $SNAPS/$1_$(date +%F_%T.%N)

num_snaps=$(ls -1d $SNAPS/$1_* | sort | wc -l)

if [ "$num_snaps" -gt "$2" ]; then
let over=$num_snaps-$2
ls -1d $SNAPS/$1_* | head -n $over | while read s; do
btrfs subvolume delete $s
done
fi

Here's my crontab:
0 * * * * /usr/local/bin/snapshot hourly 6
0 0 * * * /usr/local/bin/snapshot daily 7
0 0 * * 0 /usr/local/bin/snapshot weekly 4

-=[dave]=-

Entropy isn't what it used to be.

and much easier than trying to get snapper to compile on fedora libblocxx ?
:-)
Ken

Ken Anderson

but less powerful:-)

Ah, sure. Sorry. Packages for "blocxx" for:

	Fedora_14 	Fedora_15 =09
	RHEL-5		RHEL-6 =09
	SLE_11_SP1 =09
	openSUSE_11.4	openSUSE_Factory

are available in the openSUSE buildservice at:

http://download.opensuse.org/repositories/home:/mge1512:/snapper/

See also the build project at:

	https://build.opensuse.org/project/show?project=3Dhome%3Amge1512%3Asna=

pper

If someone sends the packaging files for Debian/Ubuntu, I can also add those.

so long -
MgE

P.S.: I also added "snapper" itself there. I am not sure though,
if it will build out of the box. ... Stay tuned.

--=20
Matthias G. Eckermann Senior Product Manager SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH Maxfeldstra=C3=9Fe 5 90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3= =BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Ken and all,

On 2011-08-17 T 19:38 +0200 Matthias G. Eckermann wrote:
=20

=20
A dinner later, the packages (.rpm/.src.rpm) for blocxx and also snapper are available in the openSUSE Buildservice at:
http://download.opensuse.org/repositories/home:/mge1512:/snapper/
for=20
Fedora_15 RHEL-6 SLE_11_SP1 openSUSE_Factory

The download directory is a YUM repository and should easily work with either of the distributions.

Enjoy!

so long -
MgE

P.S.: Again, if someone sends the packaging files for
Debian/Ubuntu, I can also add those. Unfortunately my .deb experience is limited, ...

--=20
Matthias G. Eckermann Senior Product Manager SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH Maxfeldstra=C3=9Fe 5 90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3= =BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

2:/snapper/

Hi Matthias,

I'm testing your packages on top of RHEL6 + kernel 3.2.7. A small suggestion, you should include /etc/sysconfig/snapper in the package (at least for RHEL6, haven't tested the other ones). Even if it just contains

SNAPPER_CONFIGS=3D""

Thanks,

=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

512:/snapper/

Hi Fajar,

thanks for reporting that issue, I have fixed it now.

We have now created a project in the openSUSE buildservice were we provide snapper packages for various distributions, e.g. RHEL6 and Fedora 16. Please find the downloads at:

http://download.opensuse.org/repositories/filesystems:/snapper/

I'll also add a link from the snapper home page:

http://en.opensuse.org/Portal:Snapper.

I have tested snapper on Fedora 16 and found no problems.

Regards,
Arvin

--=20
Arvin Schnell, <[email protected]>
Senior Software Engineer, Research & Development
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend=F6= rffer, HRB 16746 (AG N=FCrnberg)
Maxfeldstra=DFe 5
90409 N=FCrnberg
Germany
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

1512:/snapper/

Great! Thanks.

When I installed it back then, the first thing that comes to mind was "there's no documentation on how to get started".

http://en.opensuse.org/openSUSE:Snapper_Tutorial is good, but that' is assuming root is btrfs, and snapper is already configured to snapshot root. For other distros, you need to first create the config manually, e.g. as shown for home in http://en.opensuse.org/openSUSE:Snapper_FAQ

Could you update the tutorial, or perhaps create a new "quickstart" page? I'm kinda reluctant to do it myself since I don't use opensuse, and some of my edits might not reflect the "correct" way to do it in opensuse. If that's not possible, I'll put up the documentation somewhere else (perhaps the semi-official http://btrfs.ipv5.de/ , or my own wiki).

Two other things that I have't find is:
- how to add pre and post hooks, so (for example) snapper could create the same pre-post snapshot whenever user runs "yum", similar to when a user runs "yast" in opensuse,
- whether a rollback REALLY rolls back everyting (including binary and new/missing files), or is it git-like behavior, or if it only process text files.

=2E.. but those two aren't as important as the getting-started document= ation.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

On one machine I make hourly snapshots of my /home and of the root FS as well.

The tricky part is actually not the snapshotting, but the deletion of outdated snapshots.

That's due to the unfortunate fact (bug?), that snapshot-directories do not have their ctime set correctly at all, they have some totally bogus ctime instead.

.........../snaps/$ ls -la --time=ctime | tail

dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-05@02-31-51
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-06@02-49-29
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-07@00-17-40
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-08@01-53-29
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-10@03-09-32
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-12@00-26-54
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-13@01-40-19
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-14@04-22-07
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-15@02-29-13
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-16@10-27-57

As you can see I have to store creation date/time in the snapshot name, and then parse it out to delete snapshots e.g. older than 3 months.

So until someone cares about snapshot ctime enough to fix this, btrfs will not be a convenient FS to work with timed snapshotting/cleanup.

With respect,
Roman

Isn't the ctime the creation date of the original folder?

:

s will not be a convenient FS to work with timed snapshotting/cleanup.

It will be a nice thing to have the snapshot time returned by btrfs-prog.

something like
btrfs get stime <snapshot>

Cheers, Anand
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

<snip>

Is there some reason not to use the format used by shadow_copy2 overlay for Samba? (The one providing Shadow Volume Copy functionality for Windows clients):

Current date in this format you get like this:

@GMT-date -u '+%Y.%m.%d-%H.%M.%S'

For example: @GMT-2012.02.23-10.34.32

This way, when the volume is exported using Samba, you can easily export past copies too, without creating links.

Regards,

Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

autosnap code is available either end of this week or early next week and what you will notice is autosnap snapshots are named using uuid.

Main reason to drop time-stamp based names is that,
- test (clicking on Take-snapshot button) which took more
than one snapshot per second was failing.
- a more descriptive creation time is available using a
command line option as in the example below.
-----

 # btrfs su list -t tag=@minute,parent=/btrfs/sv1 /btrfs
 /btrfs/.autosnap/6c0dabfa-5ddb-11e1-a8c1-0800271feb99 Thu Feb 23 13:01:18 2012 /btrfs/sv1 @minute
 /btrfs/.autosnap/5669613e-5ddd-11e1-a644-0800271feb99 Thu Feb 23 13:15:01 2012 /btrfs/sv1 @minute
 -----

As of now code for time-stamp as autosnap snapshot name is commented out, if more people wanted it to be a time-stamp based names, I don't mind having that way. Please do let me know.

Thanks, Anand

On Thu, Feb 23, 2012 at 7:02 PM, Anand Jain <[email protected]> wro= te:

I thought you stopped working on this :D

Alternatives are good though. Will test yours when it's out.

=46WIW, I also have another one, based on zfsonlinux's autosnapshot scr= ipt :D

3:01:18

3:15:01

=46or me the main bonus point of having timestamp in names in the abiility to sort it by creation date by simply using "ls". As for the more-than-one-click-per-second problem, in my script I simply let it fail and return informative-enough error message.

A workaround would be adding nanosecond timestamp, or put the UUID AFTER the time stamp, e.g:
/btrfs/.autosnap/@minute_20120223_131501_123456_5669613e-5ddd-11e1-a644= -0800271feb99

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello all,

ore

a =C2=A0

23 13:01:18 2012 /btrfs/sv1 @minute

23 13:15:01 2012 /btrfs/sv1 @minute

44-0800271feb99

I wonder, if this is not mixing several aspects / requirements:
1. unique snapshot numbering
2. storing the time stamp of a snapshot
3. human visible order of snapshots

Thus we have chosen a different solution in our "snapper" approach (as announced here in Aug 2011):
- the snapshot number is a simple integer
-> human visible order of snapshots
- all metadata is stored in an XML file associated to this snapshot
- this requires that the btrfs snapshot itself is in a subdirectory of the snapshot-numbered directory.

Example:

----------------------------< snip >----------------------------

2012-02-23 15:02 (0) ~
naxos root (5) # tree -x /.snapshots/6/
/.snapshots/6/
|-- filelist-5.txt
|-- info.xml
`-- snapshot

2012-02-23 15:02 (0) ~
naxos root (5) # cat /.snapshots/6/info.xml
<?xml version=3D"1.0"?>
<snapshot>
<type>post</type>
<num>6</num>
<date>2012-02-23 11:32:59</date>
<pre_num>5</pre_num>
<cleanup>number</cleanup>
</snapshot>

----------------------------< snap >----------------------------

With introducing a store for meta information, the requirements listed above (1-3) can be fulfilled.

Shouldn't the autosnap implementation include a similar option and/or functionality? Or does it already?

=46eel free to copy what snapper does:)

so long -
MgE

P.S.: For those interested in the details: snapshots 5 and 6 are a pre-post-pair, which enclose the real action: a timezone change. "filelist-5.txt" contains the file which differ between 5 and 6 for faster access from the management layer.

--=20
Matthias G. Eckermann Senior Product Manager SUSE=C2=AE Linux Ent=
erprise
Phone: +49 30 44315731 Mobile: +49 179 2949448 Twitter: mge15=
12
SUSE LINUX Products GmbH Maxfeldstra=C3=9Fe 5 90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3= =BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'd say, that having it as configure option (do Samba-style snapshot naming vs. uuid based) would be sufficient. The question remains what should be the default.

That being said, what use-case would require snapshots taken more often than every second? I doubt that you actually can do snapshots every second on a busy file system, let alone more often. On lightly-used one they will be identical and just clutter the name-space.

Regards,

Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

Thanks for the inputs. there is no clear winner as of now. Let me keep the uuid for now, if more sysadmin feel timestamp is better we could device it that way.

-Anand

I'd like to vote for timestamp/timestamp-uuid as a sysadmin. The timestamp allows for easy conversion from clients' wants to actual commands: "I need my data from two days ago" is easy when I have timestamps to use.