netapp-alike snapshots?

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly). Example:

framstag@fex:/sw/share: ll .snapshot/

drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

I would like to have something similar with btrfs. Programming such a feature is not a general problem for me, but I think I am not the first one who wants this kind of auto-snapshooting. Is there (where?) such a tool?

I know snapper, but it has a totally different approach.

--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK

Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

REF:<[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Is use: https://github.com/jf647/btrfs-snap

2017-08-22 15:22 GMT+02:00 Ulli Horlacher <[email protected]>:

Quoted 18 lines

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly). Example:

framstag@fex:/sw/share: ll .snapshot/

drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

Quoted 21 lines

Is use: https://github.com/jf647/btrfs-snap

2017-08-22 15:22 GMT+02:00 Ulli Horlacher <[email protected]>:

Quoted 18 lines

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly). Example:

framstag@fex:/sw/share: ll .snapshot/

drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

btrfs-snap does not create local .snapshot/ sub-directories, but saves the snapshots in the toplevel root volume directory.

--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK

Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/

REF:<CAEtw4r1cDuE0F_MXe6mHDUr03BhEueBWeq7YaRYjNhhf=UD6KQ@mail.gmail.com> --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

This is possible. Use the -b or -B option.

-b basedir places the snapshot in basedir with a directory structure that mimics the mountpoint
-B basedir places the snapshots in basedir with NO additional subdirectory structure

2017-08-22 16:24 GMT+02:00 Ulli Horlacher <[email protected]>:

Quoted 25 lines · Peter Becker

Quoted 21 lines

Is use: https://github.com/jf647/btrfs-snap

2017-08-22 15:22 GMT+02:00 Ulli Horlacher <[email protected]>:

Quoted 18 lines

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly). Example:

framstag@fex:/sw/share: ll .snapshot/

drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

btrfs-snap does not create local .snapshot/ sub-directories, but saves the snapshots in the toplevel root volume directory.

Quoted 21 lines · Peter Becker

Is use: https://github.com/jf647/btrfs-snap

2017-08-22 15:22 GMT+02:00 Ulli Horlacher <[email protected]>:

Quoted 18 lines

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly). Example:

framstag@fex:/sw/share: ll .snapshot/

drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-15_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-16_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-17_0010
drwxr-xr-x  framstag root - 2017-08-14 10:21:47 .snapshot/daily.2017-08-18_0010
drwxr-xr-x  framstag root - 2017-08-18 23:59:29 .snapshot/daily.2017-08-19_0010
drwxr-xr-x  framstag root - 2017-08-19 21:01:25 .snapshot/daily.2017-08-20_0010
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/daily.2017-08-21_0010
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1210
drwxr-xr-x  framstag root - 2017-08-20 02:50:18 .snapshot/hourly.2017-08-20_1610
drwxr-xr-x  framstag root - 2017-08-20 19:48:40 .snapshot/hourly.2017-08-20_2010
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_0810
drwxr-xr-x  framstag root - 2017-08-21 00:42:28 .snapshot/hourly.2017-08-21_1210
drwxr-xr-x  framstag root - 2017-08-21 13:05:28 .snapshot/hourly.2017-08-21_1610

btrfs-snap does not create local .snapshot/ sub-directories, but saves the snapshots in the toplevel root volume directory.

It is beneficial to not have snapshots in-place. With a local directory of snapshots, issuing things like "find", "grep -r" or even "du" will take an inordinate amount of time and will produce a result you do not expect.

Personally I prefer to have a /snapshots directory on every FS

Quoted 4 lines

It is beneficial to not have snapshots in-place. With a local directory of snapshots, issuing things like "find", "grep -r" or even "du" will take an inordinate amount of time and will produce a result you do not expect.

Netapp snapshots are invisible for tools doing opendir()/readdir() One could simulate this with symlinks for the snapshot directory:
store the snapshot elsewhere (not inplace) and create a symlink to it, in every directory.

Quoted 1 line

Personally I prefer to have a /snapshots directory on every FS

My users want the snapshots locally in a .snapshot subdirectory. Because Netapp do it this way - for at least 20 years and we have a multi-PB Netapp storage environment.
No chance to change this.

Signature · 10 lines

Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20170822214531.44538589@natsu>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Perhaps using a bind mount? It would look and work the same as a ordinary fs. Just need to make sure du uses one filesystem.

---- From: Ulli Horlacher <[email protected]> -- Sent: 2017-08-22 - 18:57 ----

Quoted 10 lines · Roman Mamedov

Quoted 4 lines

It is beneficial to not have snapshots in-place. With a local directory of snapshots, issuing things like "find", "grep -r" or even "du" will take an inordinate amount of time and will produce a result you do not expect.

Netapp snapshots are invisible for tools doing opendir()/readdir() One could simulate this with symlinks for the snapshot directory:
store the snapshot elsewhere (not inplace) and create a symlink to it, in every directory.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

It is beneficial to not have snapshots in-place. With a local directory of snapshots, [ ... ]

Quoted 4 lines

Quoted 4 lines

Netapp snapshots are invisible for tools doing opendir()/ readdir() One could simulate this with symlinks for the snapshot directory: store the snapshot elsewhere (not inplace) and create a symlink to it, in every directory.

Quoted 2 lines

Quoted 2 lines

My users want the snapshots locally in a .snapshot subdirectory.

Quoted 3 lines

Quoted 3 lines

Because Netapp do it this way - for at least 20 years and we have a multi-PB Netapp storage environment. No chance to change this.

Not only du works recursivly, but also find and with option also ls, grep, etc.

And it would require a bind mount for EVERY directory. There can be hundreds... thousends!

Indeed and there is a fair description of some options for subvolume nesting policies here which may be interesting to the original poster:

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Layout

It is unsurprising to me that there are tradeoffs involved in every choice. I find the "Flat" layout particularly desirable.

Btrfs snapshots can only be done for a whole subvolume.

Subvolumes and snapshots can be created by users, but too many snapshots (see below) can cause trouble. For somewhat good reasons subvolumes including snapshots cannot be deleted by users though unless mount option 'user_subvol_rm_allowed' is used.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Because Netapp do it this way - for at least 20 years and we have a multi-PB Netapp storage environment. No chance to change this.

Send patches :-).

Assumptions that all Btrfs features such as snapshots are infinitely scalable at no cost may be optimistic:

https://btrfs.wiki.kernel.org/index.php/Gotchas#Having_many_subvolumes_can_be_very_slow

Quoted 4 lines

Assumptions that all Btrfs features such as snapshots are infinitely scalable at no cost may be optimistic:

https://btrfs.wiki.kernel.org/index.php/Gotchas#Having_many_subvolumes_can_be_very_slow

"when you do device removes on file systems with a lot of snapshots, it
is unbelievably slow ... took nearly a week to move 20GB of FS data from one device to the other using that method"

"a balance on 2TB of data that was heavily snapshotted - it took 3 months"

Quoted 11 lines

Quoted 5 lines

Assumptions that all Btrfs features such as snapshots are infinitely scalable at no cost may be optimistic:

https://btrfs.wiki.kernel.org/index.php/Gotchas#Having_many_subvolumes_can_be_very_slow

"when you do device removes on file systems with a lot of snapshots, it
is unbelievably slow ... took nearly a week to move 20GB of FS data from one device to the other using that method"

"a balance on 2TB of data that was heavily snapshotted - it took 3 months"

This is a vanilla SLES12 installation:

root@ptm1:~# grep PRETTY_NAME /etc/os-release 
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP1"

root@ptm1:~# btrfs subvolume list /
ID 257 gen 358277 top level 5 path @
ID 258 gen 978361 top level 257 path @/home
ID 259 gen 1252501 top level 257 path @/opt
ID 260 gen 883012 top level 257 path @/srv
ID 261 gen 1252673 top level 257 path @/tmp
ID 262 gen 1252501 top level 257 path @/usr/local
ID 263 gen 882958 top level 257 path @/var/crash
ID 264 gen 1252673 top level 257 path @/var/log
ID 265 gen 882923 top level 257 path @/var/opt
ID 266 gen 1252673 top level 257 path @/var/spool
ID 267 gen 1252668 top level 257 path @/var/tmp
ID 270 gen 1252668 top level 257 path @/.snapshots
ID 452 gen 358277 top level 270 path @/.snapshots/127/snapshot
ID 453 gen 1252670 top level 270 path @/.snapshots/128/snapshot
ID 540 gen 368554 top level 270 path @/.snapshots/191/snapshot
ID 542 gen 419566 top level 270 path @/.snapshots/192/snapshot
ID 1035 gen 1027889 top level 270 path @/.snapshots/539/snapshot
ID 1036 gen 1027889 top level 270 path @/.snapshots/540/snapshot
ID 1045 gen 1048327 top level 270 path @/.snapshots/545/snapshot
ID 1046 gen 1048327 top level 270 path @/.snapshots/546/snapshot
ID 1062 gen 1068800 top level 270 path @/.snapshots/555/snapshot
ID 1063 gen 1068800 top level 270 path @/.snapshots/556/snapshot
ID 1122 gen 1130369 top level 270 path @/.snapshots/595/snapshot
ID 1123 gen 1130369 top level 270 path @/.snapshots/596/snapshot
ID 1124 gen 1171229 top level 270 path @/.snapshots/597/snapshot
ID 1125 gen 1171229 top level 270 path @/.snapshots/598/snapshot
ID 1135 gen 1171229 top level 270 path @/.snapshots/605/snapshot
ID 1136 gen 1171229 top level 270 path @/.snapshots/606/snapshot
ID 1137 gen 1171229 top level 270 path @/.snapshots/607/snapshot
ID 1138 gen 1171229 top level 270 path @/.snapshots/608/snapshot
ID 1139 gen 1171229 top level 270 path @/.snapshots/609/snapshot
ID 1140 gen 1171229 top level 270 path @/.snapshots/610/snapshot
ID 1141 gen 1171229 top level 270 path @/.snapshots/611/snapshot
ID 1142 gen 1171229 top level 270 path @/.snapshots/612/snapshot
ID 1158 gen 1172970 top level 270 path @/.snapshots/613/snapshot
ID 1159 gen 1172972 top level 270 path @/.snapshots/614/snapshot

Why does SUSE ignore this "not too many subvolumes" warning?

Signature · 11 lines

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

---- From: Ulli Horlacher <[email protected]> -- Sent:
2017-08-23 - 09:18 ----

Quoted 6 lines · Ulli Horlacher

Quoted 5 lines

Quoted 5 lines

Assumptions that all Btrfs features such as snapshots are infinitely scalable at no cost may be optimistic:

https://btrfs.wiki.kernel.org/index.php/

Quoted 49 lines

Quoted 8 lines

"when you do device removes on file systems with a lot of snapshots, it
is unbelievably slow ... took nearly a week to move 20GB of FS data from one device to the other using that method"

"a balance on 2TB of data that was heavily snapshotted - it took 3 months"

This is a vanilla SLES12 installation:

root@ptm1:~# grep PRETTY_NAME /etc/os-release PRETTY_NAME="SUSE Linux
Enterprise Server 12 SP1"

root@ptm1:~# btrfs subvolume list /
ID 257 gen 358277 top level 5 path @
ID 258 gen 978361 top level 257 path @/home ID 259 gen 1252501 top
level 257 path @/opt ID 260 gen 883012 top level 257 path @/srv ID 261
gen 1252673 top level 257 path @/tmp ID 262 gen 1252501 top level 257
path @/usr/local ID 263 gen 882958 top level 257 path @/var/crash ID
264 gen 1252673 top level 257 path @/var/log ID 265 gen 882923 top
level 257 path @/var/opt ID 266 gen 1252673 top level 257 path
@/var/spool ID 267 gen 1252668 top level 257 path @/var/tmp ID 270 gen
1252668 top level 257 path @/.snapshots ID 452 gen 358277 top level 270
path @/.snapshots/127/snapshot ID 453 gen 1252670 top level 270 path
@/.snapshots/128/snapshot ID 540 gen 368554 top level 270 path
@/.snapshots/191/snapshot ID 542 gen 419566 top level 270 path
@/.snapshots/192/snapshot ID 1035 gen 1027889 top level 270 path
@/.snapshots/539/snapshot ID 1036 gen 1027889 top level 270 path
@/.snapshots/540/snapshot ID 1045 gen 1048327 top level 270 path
@/.snapshots/545/snapshot ID 1046 gen 1048327 top level 270 path
@/.snapshots/546/snapshot ID 1062 gen 1068800 top level 270 path
@/.snapshots/555/snapshot ID 1063 gen 1068800 top level 270 path
@/.snapshots/556/snapshot ID 1122 gen 1130369 top level 270 path
@/.snapshots/595/snapshot ID 1123 gen 1130369 top level 270 path
@/.snapshots/596/snapshot ID 1124 gen 1171229 top level 270 path
@/.snapshots/597/snapshot ID 1125 gen 1171229 top level 270 path
@/.snapshots/598/snapshot ID 1135 gen 1171229 top level 270 path
@/.snapshots/605/snapshot ID 1136 gen 1171229 top level 270 path
@/.snapshots/606/snapshot ID 1137 gen 1171229 top level 270 path
@/.snapshots/607/snapshot ID 1138 gen 1171229 top level 270 path
@/.snapshots/608/snapshot ID 1139 gen 1171229 top level 270 path
@/.snapshots/609/snapshot ID 1140 gen 1171229 top level 270 path
@/.snapshots/610/snapshot ID 1141 gen 1171229 top level 270 path
@/.snapshots/611/snapshot ID 1142 gen 1171229 top level 270 path
@/.snapshots/612/snapshot ID 1158 gen 1172970 top level 270 path
@/.snapshots/613/snapshot ID 1159 gen 1172972 top level 270 path
@/.snapshots/614/snapshot

Quoted 1 line

Why does SUSE ignore this "not too many subvolumes" warning?

Using hundreds or thousands of snapshots is probably fine mostly. perhaps the slow performance is more related to what changed between them? I have regularly that many snapshots and export many as "Previous Versions" to Windows clients over Samba without any performance issues. But my data doesn't change that much.

I think those comments on the Wiki are a little misleading without better details to what workloads are affected this way. Perhaps someone can set up some tests and publish the results?

Quoted 12 lines

--
Ullrich Horlacher Server und Virtualisierung Rechenzentrum
TIK Universitaet Stuttgart E-Mail:
[email protected] Allmandring 30a Tel:
++49-711-68565868 70569 Stuttgart (Germany) WWW:
http://www.tik.uni-stuttgart.de/
REF:<[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 2 lines

Using hundreds or thousands of snapshots is probably fine mostly.

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

As usual with Btrfs, there are corner cases to avoid: 'defrag' should be done before 'balance'

and with compression switched off

I prefer dump-and-reload.

Quoted 2 lines · Peter Grandi

As usual with Btrfs, there are corner cases to avoid: 'defrag' should be done before 'balance'

Good hint. So far I did it the other way: balance before defrag. I will switch.

Quoted 3 lines

and with compression switched off

I have filesystems with compress mount option:

framstag@fex:~: grep /local /etc/fstab
LABEL=local /local btrfs defaults,compress,user_subvol_rm_allowed 0 2

and a weekly cronjob, which does a defrag and balance. I cannot disable compression.
Any hint here?

Quoted 3 lines

I prefer dump-and-reload.

What do you mean by this?

[...]

Quoted 1 line

Any hint here?

Having compression enabled causes no issues with defray and balance. There appears to be a prevalent belief however that defrag is pointless if you're using compression, probably because some versions of filefrag don't report compressed extents properly (they list each 128k compressed unit as one extent, which is wrong).

Quoted 7 lines

[...]

Quoted 1 line

Any hint here?

Having compression enabled causes no issues with defray and balance. There appears to be a prevalent belief however that defrag is pointless if you're using compression, probably because some versions of filefrag don't report compressed extents properly (they list each 128k compressed unit as one extent, which is wrong).

Is there another tool to verify fragments number of given file when using compression?

Michał Sokołowski posted on Thu, 31 Aug 2017 16:38:14 +0200 as excerpted:

Quoted 2 lines

Is there another tool to verify fragments number of given file when using compression?

AFAIK there isn't an official one, tho someone posted a script (python, IIRC) at one point and may repost it here.

You can actually get the information needed from filefrag -v (and the script does), but it takes a bit more effort than usual, scripted or brain-power, to convert the results into real fragmentation numbers.

The problem is that btrfs compression works in 128 KiB blocks, and filefrag sees each of those as a fragment. The extra effort involves checking the addresses of the reported 128 KiB blocks to see if they are actually contiguous, that is, one starts just after the previous one ends. If so it's actually not fragmented at that point. But if the addresses aren't contiguous, there's fragmentation at that point.

I don't personally worry too much about it here, for two reasons. First, I /always/ run with the autodefrag mount option, which keeps fragmentation manageable in any case[1], and second, I'm on ssd, where the effects of fragmentation aren't as pronounced. (On spinning rust it's generally the seek times that dominate. On ssds that's 0, but there's still an IOPS cost.)

So while I've run filefrag -v and looked at the results a few times out of curiousity, and indeed could see the reported fragmentation that was actually contiguous, it was simply a curiosity to me, thus my not grabbing that script and putting it to immediate use.

---
[1] AFAIK autodefrag checks fragmentation on writes, and rewrites 16 MiB blocks if necessary. If like me you always run it from the moment you start putting data on the filesystem, that should work pretty well. If however you haven't been running it or doing manual defrag, because defrag only works on writes and the free space may be fragmented enough there's not 16 MiB blocks to write into, it may take awhile to "catch up", and of course won't defrag anything that's never written to again, but is often reread, making its existing fragmentation an issue.

Very comprehensive, thank you. I was asking because I'd like to learn how really random writes by VM affects BTRFS (vs XFS,Ext4) performance and try to develop some workaround to reduce/prevent it while having csums, cow (snapshots) and compression.

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

Quoted 6 lines

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

For meaningful discussion it is important to take into account the fact

that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work.

I've got two near-identical servers here with several containers each different only on in filesystem: btrfs-raid1 on one (for historical reasons) and ext4/mdadm-raid1 on another, no snapshots, no reflinks. Each time containers on ext4 update several times faster, but in everyday operation there's no significant difference.
--

With Best Regards,
Marat Khalili

Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili:

Quoted 8 lines

Quoted 6 lines

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

For meaningful discussion it is important to take into account the fact

Doing daily updates on a desktop is not uncommon and when 3 minutes become 30 then many would call that meaningfull.

Similar for a single office server, which is upgraded twice a year, where an upgrade normally would take an hour or 2, but now more than a working day. In the meantime, take samba and postgresql offline, preventing people to work for a few hours.

My point is: fsync is not targeted specifically in many common disk bench marks (phoronix?), it might be posible that there is no trigger to spend much time on optimizations in that area. That doesn't make it meaningless.

Quoted 3 lines

that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work.

OTOH it would be nice if dpkg would at last start making use btrfs snapshot features and abandon these unnecssary fsyncs completely, instead restoring a failed install from a snapshot. This would probably result in a performance improve compared to ext4.

Quoted 5 lines

I've got two near-identical servers here with several containers each different only on in filesystem: btrfs-raid1 on one (for historical reasons) and ext4/mdadm-raid1 on another, no snapshots, no reflinks. Each time containers on ext4 update several times faster, but in everyday operation there's no significant difference.

Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili:

Quoted 4 lines

Quoted 4 lines

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Quoted 4 lines

Quoted 2 lines

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

For meaningful discussion it is important to take into account the fact

Doing daily updates on a desktop is not uncommon and when 3 minutes become 30 then many would call that meaningful.

Similar for a single office server, which is upgraded twice a year, where an upgrade normally would take an hour or 2, but now more than a working day. In the meantime, take samba and postgresql offline, preventing people to work for a few hours.

My point is: fsync is not targeted specifically in many common disk bench marks (phoronix?), it might be posible that there is no trigger to spend much time on optimizations in that area. That doesn't make it meaningless.

Quoted 3 lines

that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work.

OTOH it would be nice if dpkg would at last start making use btrfs snapshot features and abandon these unnecssary fsyncs completely, instead restoring a failed install from a snapshot. This would probably result in a performance improve compared to ext4.

Quoted 9 lines

Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili:

Quoted 7 lines

Quoted 6 lines

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

Quoted 5 lines

Quoted 2 lines

For meaningful discussion it is important to take into account the fact

Doing daily updates on a desktop is not uncommon and when 3 minutes become 30 then many would call that meaningful.

I think the more meaningful aspect here is that it's 30 minutes where persistent storage is liable to be unusable, not necessarily that it's 30 minutes.

Quoted 4 lines

Similar for a single office server, which is upgraded twice a year, where an upgrade normally would take an hour or 2, but now more than a working day. In the meantime, take samba and postgresql offline, preventing people to work for a few hours.

That should only be the case if:
1. You don't have your data set properly segregated from the rest of your system (it should not be part of the upgrade snapshot, but an independent snapshot taken separately).
2. You are updating the main system, instead of updating the snapshot you took.

The ideal method of handling an upgrade in this case is:
1. Snapshot the system, but not the data set.
2. Run your updates on the snapshot of the system.
3. Rename the snapshot and the root subvolume so that you boot into the snapshot.
4. During the next maintenance window (or overnight), shutdown the system services, snapshot the data set (so you can roll back if the update screws up the database).
5. Reboot.

That provides minimal downtime, and removes the need to roll-back if the upgrade fails part way through (you just nuke the snapshot and start over, instead of having to manually switch to the snapshot and reboot).

Quoted 10 lines

My point is: fsync is not targeted specifically in many common disk bench marks (phoronix?), it might be posible that there is no trigger to spend much time on optimizations in that area. That doesn't make it meaningless.

Quoted 4 lines

that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work.

Quoted 4 lines

OTOH it would be nice if dpkg would at last start making use btrfs snapshot features and abandon these unnecssary fsyncs completely, instead restoring a failed install from a snapshot. This would probably result in a performance improve compared to ext4.

Not dpkg, apt-get and wherever other frontedd you use (although all the other dpkg frontends I know of are actually apt-get frontends). Take a look at how SUSE actually does this integration, it's done through Zypper/YaST2, not RPM. If you do it through dpkg, or RPM, or whatever other low-level package tool, you need to do a snapshot per package so that it works reliably, while what you really need is a snapshot per high-level transaction.

FWIW, if you can guarantee that the system won't crash during an update (or are actually able to roll back by hand easily if it won't boot), you can install libeatmydata and LD_PRELOAD it for the apt-get (or aptitude, or synaptic, or whatever else) call, then call sync afterwards and probably see a significant perofrmance improvement. The library itself overloads *sync() calls to be no-ops, so it's not safe to use when you don't have good fallback options, but it tends to severely improve performance for stuff like dpkg.

Op Fri, 25 Aug 2017 07:45:44 -0400, schreef Austin S. Hemmelgarn:

Quoted 13 lines · Ferry Toth

Quoted 12 lines

Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili:

Quoted 9 lines

Quoted 6 lines

We find that typically apt is very slow on a machine with 50 or so snapshots and raid10. Slow as in probably 10x slower as doing the same update on a machine with 'single' and no snapshots.

Other operations seem to be the same speed, especially disk benchmarks do not seem to indicate any performance degradation.

For meaningful discussion it is important to take into account the fact

Quoted 5 lines

Quoted 2 lines

Doing daily updates on a desktop is not uncommon and when 3 minutes become 30 then many would call that meaningful.

I think the more meaningful aspect here is that it's 30 minutes where persistent storage is liable to be unusable, not necessarily that it's 30 minutes.

Unusable - probably less (depending on your definition)

Irritating - yes

Quoted 22 lines

Quoted 5 lines

Similar for a single office server, which is upgraded twice a year, where an upgrade normally would take an hour or 2, but now more than a working day. In the meantime, take samba and postgresql offline, preventing people to work for a few hours.

That should only be the case if:
1. You don't have your data set properly segregated from the rest of your system (it should not be part of the upgrade snapshot, but an independent snapshot taken separately).
2. You are updating the main system, instead of updating the snapshot you took.

The ideal method of handling an upgrade in this case is:
1. Snapshot the system, but not the data set.
2. Run your updates on the snapshot of the system.
3. Rename the snapshot and the root subvolume so that you boot into the snapshot.
4. During the next maintenance window (or overnight), shutdown the system services, snapshot the data set (so you can roll back if the update screws up the database).
5. Reboot.

Quoted 3 lines

That provides minimal downtime, and removes the need to roll-back if the upgrade fails part way through (you just nuke the snapshot and start over, instead of having to manually switch to the snapshot and reboot).

Wow, yes that does sound ideal. Is that how you do it?

Now I just need Cananonical to update there installer to take of this. (that is: tell it to update the system on another partition (subvolume) than the one mounted on /, and not stop any running system services).

Or run a virtual machine on the server that boots from the snapshot and that update. Oh no, the virtual machine would slow down my running server to much.

Eh, share the snapshot via cifs or nfs to another machine that does a netboot and let that do the update.

O wait, I forgot, I installed btrfs to make our system maintenance easier than with ext. Maybe that was a mistake, at least until distros take advantage of the aadvantage and start avoiding the pitfalls?

Quoted 23 lines

Quoted 11 lines

My point is: fsync is not targeted specifically in many common disk bench marks (phoronix?), it might be posible that there is no trigger to spend much time on optimizations in that area. That doesn't make it meaningless.

Quoted 4 lines

that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work.

Quoted 4 lines

OTOH it would be nice if dpkg would at last start making use btrfs snapshot features and abandon these unnecssary fsyncs completely, instead restoring a failed install from a snapshot. This would probably result in a performance improve compared to ext4.

Not dpkg, apt-get and wherever other frontedd you use (although all the other dpkg frontends I know of are actually apt-get frontends). Take a look at how SUSE actually does this integration, it's done through Zypper/YaST2, not RPM. If you do it through dpkg, or RPM, or whatever other low-level package tool, you need to do a snapshot per package so that it works reliably, while what you really need is a snapshot per high-level transaction.

Quoted 8 lines

FWIW, if you can guarantee that the system won't crash during an update (or are actually able to roll back by hand easily if it won't boot), you can install libeatmydata and LD_PRELOAD it for the apt-get (or aptitude, or synaptic, or whatever else) call, then call sync afterwards and probably see a significant perofrmance improvement. The library itself overloads *sync() calls to be no-ops, so it's not safe to use when you don't have good fallback options, but it tends to severely improve performance for stuff like dpkg.

Yeah I can guarantee that it can crash... All you need to do is start the upgrade from a remote terminal, forget to use screen and then close the terminal. And if not the installer will run until the first 'do you want to replace the conf file Y/n' in the background, at which point you have no choice than to nuke it.

But probably if you take a snapshot before eating the data you should be able to recover.

This is a vanilla SLES12 installation: [ ... ] Why does SUSE ignore this "not too many subvolumes" warning?

For somewhat good reasons subvolumes including snapshots cannot be deleted by users though unless mount option 'user_subvol_rm_allowed' is used.

Also in https://btrfs.wiki.kernel.org/index.php/Mount_options
"user_subvol_rm_allowed (...) Use with caution."

Why? What is the problem?

Quoted 4 lines

Also in https://btrfs.wiki.kernel.org/index.php/Mount_options
"user_subvol_rm_allowed (...) Use with caution."

Why? What is the problem?

Because with the mount option any user can delete any subvolume, including root one (subvol_id=5)

From btrfs-progs cmds-subvolume.c:

Quoted 5 lines

Quoted 4 lines

Also in https://btrfs.wiki.kernel.org/index.php/Mount_options
"user_subvol_rm_allowed (...) Use with caution."

Why? What is the problem?

Quoted 2 lines

Because with the mount option any user can delete any subvolume, including root one (subvol_id=5)

Apologies, it works somewhat different:
filesystem doesn't allow to delete subvolume with id 5 and POSIX access is checked before deleting subvolume with user_subvol_rm_allowed mount option.

So, still: What is the problem with user_subvol_rm_allowed?

Quoted 1 line

So, still: What is the problem with user_subvol_rm_allowed?

As usual, it is complicated: mostly that while subvol creation is very cheap, subvol deletion can be very expensive. But then so can be creating many snapshots, as in this:

Quoted 5 lines · Peter Grandi

Quoted 1 line

So, still: What is the problem with user_subvol_rm_allowed?

As usual, it is complicated: mostly that while subvol creation is very cheap, subvol deletion can be very expensive. But then so can be creating many snapshots, as in this:

But it seems one cannot prohibit a user making snapshots? Then root must delete them?

Quoted 3 lines · Roman Mamedov

It is beneficial to not have snapshots in-place. With a local directory of snapshots, issuing things like "find", "grep -r" or even "du" will take an inordinate amount of time and will produce a result you do not expect.

Netapp snapshots are invisible for tools doing opendir()/readdir() One could simulate this with symlinks for the snapshot directory:
store the snapshot elsewhere (not inplace) and create a symlink to it, in every directory.

Quoted 1 line

Personally I prefer to have a /snapshots directory on every FS

My users want the snapshots locally in a .snapshot subdirectory. Because Netapp do it this way - for at least 20 years and we have a multi-PB Netapp storage environment.
No chance to change this.

Quoted 3 lines

My users want the snapshots locally in a .snapshot subdirectory. Because Netapp do it this way - for at least 20 years and we have a multi-PB Netapp storage environment.

Just a side note, you do know that only subvolumes can be snapshotted on Btrfs, not any regular directory? And that snapshots are not recursive, i.e. if a subvolume "contains" other subvolumes (hint: it really doesn't), snapshots of the parent one will not include content of subvolumes below that in the tree.

I don't know how Netapp does this

from the way you describe that setup it feels like with Btrfs you're still in for some bad surprises and a part of your expectations will not be met.

Do you plan to make each and every directory and subdirectory a subvolume

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly).

I would like to have something similar with btrfs. Is there (where?) such a tool?

Quoted 5 lines · Ulli Horlacher

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly).

I would like to have something similar with btrfs. Is there (where?) such a tool?

I have found none, so I have implemented it by myself:

https://fex.rus.uni-stuttgart.de/snaprotate.html

Quoted 4 lines · Ulli Horlacher

Quoted 2 lines

With Netapp/waffle you have automatic hourly/daily/weekly snapshots. You can find these snapshots in every local directory (readonly).

Quoted 3 lines

I have found none, so I have implemented it by myself:

https://fex.rus.uni-stuttgart.de/snaprotate.html

Not sure how you looked :)
http://marc.merlins.org/perso/btrfs/post_2014-03-21_Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots.html

Might not be exactly what you wanted, but been using it for 3 years.

Your tool does not create .snapshot subdirectories in EVERY directory like

Netapp does.
Example:

framstag@fex:~: cd ~/Mail/.snapshot/
framstag@fex:~/Mail/.snapshot: l
lR-X - 2017-09-09 09:55 2017-09-09_0000.daily -> /local/home/.snapshot/2017-09-09_0000.daily/framstag/Mail

lR-X - 2017-09-09 14:00 2017-09-09_1400.hourly -> /local/home/.snapshot/2017-09-09_1400.hourly/framstag/Mail lR-X - 2017-09-09 15:00 2017-09-09_1500.hourly -> /local/home/.snapshot/2017-09-09_1500.hourly/framstag/Mail lR-X - 2017-09-09 15:18 2017-09-09_1518.single -> /local/home/.snapshot/2017-09-09_1518.single/framstag/Mail lR-X - 2017-09-09 15:20 2017-09-09_1520.single -> /local/home/.snapshot/2017-09-09_1520.single/framstag/Mail lR-X - 2017-09-09 15:22 2017-09-09_1522.single -> /local/home/.snapshot/2017-09-09_1522.single/framstag/Mail

My users (and I) need snapshots in this way.

Quoted 1 line

Your tool does not create .snapshot subdirectories in EVERY directory like

Neither does NetApp. Those "directories" are magic handles that do not really exist.

Apart from obvious problem with recursive directory traversal (NetApp .snapshot are not visible with normal directory list)

Quoted 5 lines

Quoted 2 lines

Your tool does not create .snapshot subdirectories in EVERY directory like

Neither does NetApp. Those "directories" are magic handles that do not really exist.

I know.
But symbolic links are the next close thing (I am not a kernel programmer).

Quoted 2 lines

Apart from obvious problem with recursive directory traversal (NetApp .snapshot are not visible with normal directory list)

Yes, they are, at least sometimes, eg tar includes the snapshots.

Signature · 10 lines

Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

09.09.2017 16:44, Ulli Horlacher пишет:

Quoted 2 lines

Your tool does not create .snapshot subdirectories in EVERY directory like

Neither does NetApp. Those "directories" are magic handles that do not really exist.

Quoted 6 lines

Netapp does.
Example:

framstag@fex:~: cd ~/Mail/.snapshot/
framstag@fex:~/Mail/.snapshot: l
lR-X - 2017-09-09 09:55 2017-09-09_0000.daily -> /local/home/.snapshot/2017-09-09_0000.daily/framstag/Mail

Apart from obvious problem with recursive directory traversal (NetApp .snapshot are not visible with normal directory list) those will also be captured in snapshots and cannot be removed. NetApp snapshots themselves do not expose .snapshot "directories".

Quoted 7 lines

lR-X - 2017-09-09 14:00 2017-09-09_1400.hourly -> /local/home/.snapshot/2017-09-09_1400.hourly/framstag/Mail lR-X - 2017-09-09 15:00 2017-09-09_1500.hourly -> /local/home/.snapshot/2017-09-09_1500.hourly/framstag/Mail lR-X - 2017-09-09 15:18 2017-09-09_1518.single -> /local/home/.snapshot/2017-09-09_1518.single/framstag/Mail lR-X - 2017-09-09 15:20 2017-09-09_1520.single -> /local/home/.snapshot/2017-09-09_1520.single/framstag/Mail lR-X - 2017-09-09 15:22 2017-09-09_1522.single -> /local/home/.snapshot/2017-09-09_1522.single/framstag/Mail

My users (and I) need snapshots in this way.