Skip to contentSkip to conversation

No files in snapshot

DISCUSSION Thomas Schneider 4 participants 10 messages Aug 27, 2019 – Sep 5, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I'm running Debian 10 with btrfs-progs=5.2.1. Creating snapshots with snapper=0.8.2 works w/o errors.

root@ld5507:~# uname -r
5.0.18-1-pve
root@ld5508:~# btrfs fi show
Label: none  uuid: 70997a0b-a36f-46cf-9ddc-b88d47eabc9e
        Total devices 1 FS bytes used 11.17GiB
        devid    1 size 192.53GiB used 22.02GiB path /dev/sdbq3
root@ld5507:~# btrfs fi df /
Data, single: total=18.00GiB, used=10.71GiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=2.00GiB, used=465.84MiB
GlobalReserve, single: total=68.03MiB, used=0.00B

However, I run into an issue and need to restore various files.

I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!

Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.

These files are required:

root@ld5507:/usr/bin# ls -l /var/lib/ceph/osd/ceph-4/
insgesamt 56
-rw-r--r-- 1 root root 402 Jun  7 12:18 activate.monmap
-rw-r--r-- 1 ceph ceph   3 Jun  7 12:18 active
lrwxrwxrwx 1 ceph ceph  58 Jun  7 12:18 block -> /dev/disk/by-partuuid/3bc0c812-2c6b-4544-bbe7-e0444c3448eb
-rw-r--r-- 1 ceph ceph  37 Jun  7 12:18 block_uuid
-rw-r--r-- 1 ceph ceph   2 Jun  7 12:18 bluefs
-rw-r--r-- 1 ceph ceph  37 Jun  7 12:18 ceph_fsid
-rw-r--r-- 1 ceph ceph  37 Jun  7 12:18 fsid
-rw------- 1 ceph ceph  56 Jun  7 12:18 keyring
-rw-r--r-- 1 ceph ceph   8 Jun  7 12:18 kv_backend
-rw-r--r-- 1 ceph ceph  21 Jun  7 12:18 magic
-rw-r--r-- 1 ceph ceph   4 Jun  7 12:18 mkfs_done
-rw-r--r-- 1 ceph ceph   6 Jun  7 12:18 ready
-rw-r--r-- 1 ceph ceph   3 Aug 23 09:56 require_osd_release
-rw-r--r-- 1 ceph ceph   0 Aug 21 12:41 systemd
-rw-r--r-- 1 ceph ceph  10 Jun  7 12:18 type
-rw-r--r-- 1 ceph ceph   2 Jun  7 12:18 whoami

And there are not files in latest snapshot:

root@ld5507:/usr/bin# ls -l /btrfs/@snapshots/158/snapshot/var/lib/ceph/osd/ceph-4/
insgesamt 0

Why is this?

THX

#1 lore raw
Quoted 8 lines · Thomas Schneider

However, I run into an issue and need to restore various files.

I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!

Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.

Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.

# btrfs sub list -to /var/lib
# btrfs sub list -to /var/

In some sense these are redundant, I'm not sure if your /var/lib is a subvolume or not. Also please include the exact snapshot command you're making.

Signature · 1 line
Chris Murphy
#2 lore raw

Hi,

I was thinking of this, too. But it does not apply.

root@ld5507:~# btrfs su list -to /var/lib
ID      gen     top level       path
--      ---     ---------       ----
root@ld5507:~# btrfs su list -to /var
ID      gen     top level       path
--      ---     ---------       ----

And there are files in other directories:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/
insgesamt 4
-rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyring
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/
insgesamt 12
-rw------- 1 ceph ceph  77 Mai 28 14:33 keyring
-rw-r--r-- 1 ceph ceph   8 Mai 28 14:33 kv_backend
-rw-r--r-- 1 ceph ceph   3 Aug 23 09:41 min_mon_release
drwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db

Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/
insgesamt 0

To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"

Am 28.08.2019 um 00:24 schrieb Chris Murphy:

Quoted 19 lines · Thomas Schneider
Quoted 7 lines

However, I run into an issue and need to restore various files.

I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!

Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.

Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.

# btrfs sub list -to /var/lib
# btrfs sub list -to /var/

In some sense these are redundant, I'm not sure if your /var/lib is a subvolume or not. Also please include the exact snapshot command you're making.

#3 lore raw
Quoted 23 lines · Thomas Schneider

Hi,

I was thinking of this, too. But it does not apply.

root@ld5507:~# btrfs su list -to /var/lib
ID      gen     top level       path
--      ---     ---------       ----
root@ld5507:~# btrfs su list -to /var
ID      gen     top level       path
--      ---     ---------       ----

And there are files in other directories:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/
insgesamt 4
-rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyring
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/
insgesamt 12
(quoted diff) +0 -3
-rw------- 1 ceph ceph  77 Mai 28 14:33 keyring
-rw-r--r-- 1 ceph ceph   8 Mai 28 14:33 kv_backend
-rw-r--r-- 1 ceph ceph   3 Aug 23 09:41 min_mon_release

drwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db

Quoted 9 lines

Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/
insgesamt 0

To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

Signature · 1 line
Chris Murphy
#4 lore raw
Quoted 10 lines · Chris Murphy
Quoted 1 line

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

In this case, his snapshots are all of the root.

I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount

What is the the result of running:
mount | grep /var

#5 lore raw
Quoted 14 lines · Remi Gauvin
Quoted 10 lines · Chris Murphy
Quoted 1 line

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

Quoted 9 lines

In this case, his snapshots are all of the root.

I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount

What is the the result of running:
mount | grep /var

Yep.

Signature · 1 line
Chris Murphy
#6 lore raw

Am 04.09.19 um 21:07 schrieb Chris Murphy:

Quoted 21 lines · Remi Gauvin
Quoted 20 lines · Chris Murphy
Quoted 11 lines
Quoted 2 lines

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

In this case, his snapshots are all of the root.

I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount

Quoted 7 lines
Quoted 3 lines

What is the the result of running:
mount | grep /var

Yep.

Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.

That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).

#7 lore raw

Hi,

I would agree with Oliver's conclusion:
the relevant directories are all tmpfs mounts.

root@ld5505:~# mount | grep /var
/dev/sdbq3 on /var/cache type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=260,subvol=/@cache)
/dev/sdbq3 on /var/lib/vz/images type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=262,subvol=/@images)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /var/lib/ceph/osd/ceph-122 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-123 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-105 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-92 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-112 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-77 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-76 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-87 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-119 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-79 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-115 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-98 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-100 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-89 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-108 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-120 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-82 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-118 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-104 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-95 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-1 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-96 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-117 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-116 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-99 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-106 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-110 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-97 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-81 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-121 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-88 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-94 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-113 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-107 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-101 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-78 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-93 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-85 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-103 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-102 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-109 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-114 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-80 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-111 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-83 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-86 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-91 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-84 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-90 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-8 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-9 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-10 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-11 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-12 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-13 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-14 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-15 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-16 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-17 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-18 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-19 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-21 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-22 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-23 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-24 type tmpfs (rw,relatime)

Am 04.09.2019 um 22:52 schrieb Oliver Freyermuth:

Quoted 30 lines

Am 04.09.19 um 21:07 schrieb Chris Murphy:

Quoted 13 lines · Remi Gauvin
Quoted 12 lines · Chris Murphy
Quoted 9 lines

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

In this case, his snapshots are all of the root.

Quoted 10 lines
Quoted 7 lines

I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount

What is the the result of running:
mount | grep /var

Yep.

Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.

That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).

#8 lore raw

Dear Thomas,

Am 05.09.19 um 10:50 schrieb Thomas Schneider:

Quoted 4 lines

Hi,

I would agree with Oliver's conclusion:
the relevant directories are all tmpfs mounts.

this of course easily explains it and btrfs can't do anything about these when snapshotting.

In case you want to debug the issue in more detail (e.g. how to extract this information stored in tmpfs from LVM for example), we can of course take this to the ceph-users list (or contact me directly).

Cheers,
Oliver

Quoted 106 lines
root@ld5505:~# mount | grep /var
/dev/sdbq3 on /var/cache type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=260,subvol=/@cache)
/dev/sdbq3 on /var/lib/vz/images type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=262,subvol=/@images)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /var/lib/ceph/osd/ceph-122 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-123 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-105 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-92 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-112 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-77 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-76 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-87 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-119 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-79 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-115 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-98 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-100 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-89 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-108 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-120 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-82 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-118 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-104 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-95 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-1 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-96 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-117 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-116 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-99 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-106 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-110 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-97 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-81 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-121 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-88 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-94 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-113 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-107 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-101 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-78 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-93 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-85 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-103 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-102 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-109 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-114 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-80 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-111 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-83 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-86 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-91 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-84 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-90 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-8 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-9 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-10 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-11 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-12 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-13 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-14 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-15 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-16 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-17 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-18 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-19 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-21 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-22 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-23 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-24 type tmpfs (rw,relatime)

Am 04.09.2019 um 22:52 schrieb Oliver Freyermuth:

Quoted 30 lines

Am 04.09.19 um 21:07 schrieb Chris Murphy:

Quoted 13 lines · Remi Gauvin
Quoted 12 lines · Chris Murphy
Quoted 9 lines

I don't really know how snapper works.

The way 'btrfs subvolume snapshot' works,  you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-....  would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.

In this case, his snapshots are all of the root.

Quoted 10 lines
Quoted 7 lines

I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount

What is the the result of running:
mount | grep /var

Yep.

Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.

That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).

#9 lore raw
In reply to Chris Murphy

Can you share any update on this issue?

-------------------------------

Hi,

I was thinking of this, too. But it does not apply.

root@ld5507:~# btrfs su list -to /var/lib
ID      gen     top level       path
--      ---     ---------       ----
root@ld5507:~# btrfs su list -to /var
ID      gen     top level       path
--      ---     ---------       ----

And there are files in other directories:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/
insgesamt 4
-rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyring
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/
insgesamt 12
-rw------- 1 ceph ceph  77 Mai 28 14:33 keyring
-rw-r--r-- 1 ceph ceph   8 Mai 28 14:33 kv_backend
-rw-r--r-- 1 ceph ceph   3 Aug 23 09:41 min_mon_release
drwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db

Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:

root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/
insgesamt 0

To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"

Am 28.08.2019 um 00:24 schrieb Chris Murphy:

Quoted 19 lines · Thomas Schneider
Quoted 7 lines

However, I run into an issue and need to restore various files.

I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!

Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.

Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.

# btrfs sub list -to /var/lib
# btrfs sub list -to /var/

In some sense these are redundant, I'm not sure if your /var/lib is a subvolume or not. Also please include the exact snapshot command you're making.