No files in snapshot
Hi,
I'm running Debian 10 with btrfs-progs=5.2.1. Creating snapshots with snapper=0.8.2 works w/o errors.
root@ld5507:~# uname -r 5.0.18-1-pve root@ld5508:~# btrfs fi show Label: none uuid: 70997a0b-a36f-46cf-9ddc-b88d47eabc9e Total devices 1 FS bytes used 11.17GiB devid 1 size 192.53GiB used 22.02GiB path /dev/sdbq3
root@ld5507:~# btrfs fi df / Data, single: total=18.00GiB, used=10.71GiB System, DUP: total=8.00MiB, used=16.00KiB Metadata, DUP: total=2.00GiB, used=465.84MiB GlobalReserve, single: total=68.03MiB, used=0.00B
However, I run into an issue and need to restore various files.
I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!
Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.
These files are required:
root@ld5507:/usr/bin# ls -l /var/lib/ceph/osd/ceph-4/ insgesamt 56 -rw-r--r-- 1 root root 402 Jun 7 12:18 activate.monmap -rw-r--r-- 1 ceph ceph 3 Jun 7 12:18 active lrwxrwxrwx 1 ceph ceph 58 Jun 7 12:18 block -> /dev/disk/by-partuuid/3bc0c812-2c6b-4544-bbe7-e0444c3448eb -rw-r--r-- 1 ceph ceph 37 Jun 7 12:18 block_uuid -rw-r--r-- 1 ceph ceph 2 Jun 7 12:18 bluefs -rw-r--r-- 1 ceph ceph 37 Jun 7 12:18 ceph_fsid -rw-r--r-- 1 ceph ceph 37 Jun 7 12:18 fsid -rw------- 1 ceph ceph 56 Jun 7 12:18 keyring -rw-r--r-- 1 ceph ceph 8 Jun 7 12:18 kv_backend -rw-r--r-- 1 ceph ceph 21 Jun 7 12:18 magic -rw-r--r-- 1 ceph ceph 4 Jun 7 12:18 mkfs_done -rw-r--r-- 1 ceph ceph 6 Jun 7 12:18 ready -rw-r--r-- 1 ceph ceph 3 Aug 23 09:56 require_osd_release -rw-r--r-- 1 ceph ceph 0 Aug 21 12:41 systemd -rw-r--r-- 1 ceph ceph 10 Jun 7 12:18 type -rw-r--r-- 1 ceph ceph 2 Jun 7 12:18 whoami
And there are not files in latest snapshot:
root@ld5507:/usr/bin# ls -l /btrfs/@snapshots/158/snapshot/var/lib/ceph/osd/ceph-4/ insgesamt 0
Why is this?
THX
Quoted 8 lines · Thomas Schneider
However, I run into an issue and need to restore various files.
I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!
Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.
Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.
# btrfs sub list -to /var/lib # btrfs sub list -to /var/
In some sense these are redundant, I'm not sure if your /var/lib is a subvolume or not. Also please include the exact snapshot command you're making.
Signature · 1 line
Chris Murphy
Hi,
I was thinking of this, too. But it does not apply.
root@ld5507:~# btrfs su list -to /var/lib ID gen top level path -- --- --------- ---- root@ld5507:~# btrfs su list -to /var ID gen top level path -- --- --------- ----
And there are files in other directories:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/ insgesamt 4 -rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyring
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/ insgesamt 12 -rw------- 1 ceph ceph 77 Mai 28 14:33 keyring -rw-r--r-- 1 ceph ceph 8 Mai 28 14:33 kv_backend -rw-r--r-- 1 ceph ceph 3 Aug 23 09:41 min_mon_release drwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db
Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/ insgesamt 0
To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"
Am 28.08.2019 um 00:24 schrieb Chris Murphy:
Quoted 19 lines · Thomas Schneider
Quoted 7 lines
However, I run into an issue and need to restore various files.
I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!
Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.
Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.
# btrfs sub list -to /var/lib # btrfs sub list -to /var/In some sense these are redundant, I'm not sure if your
/var/libis a subvolume or not. Also please include the exact snapshot command you're making.
Quoted 23 lines · Thomas Schneider
Hi,
I was thinking of this, too. But it does not apply.
root@ld5507:~# btrfs su list -to /var/lib ID gen top level path -- --- --------- ---- root@ld5507:~# btrfs su list -to /var ID gen top level path -- --- --------- ----And there are files in other directories:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/ insgesamt 4 -rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyringroot@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/ insgesamt 12(quoted diff) +0
-3-rw------- 1 ceph ceph 77 Mai 28 14:33 keyring -rw-r--r-- 1 ceph ceph 8 Mai 28 14:33 kv_backend -rw-r--r-- 1 ceph ceph 3 Aug 23 09:41 min_mon_releasedrwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db
Quoted 9 lines
Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/ insgesamt 0To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or /var/lib which means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.
Signature · 1 line
Chris Murphy
Quoted 10 lines · Chris Murphy
Quoted 1 line
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or
/var/libwhich means trying to snapshot/var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.
In this case, his snapshots are all of the root.
I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount
What is the the result of running:
mount | grep /var
Quoted 14 lines · Remi Gauvin
Quoted 10 lines · Chris Murphy
Quoted 1 line
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or
/var/libwhich means trying to snapshot/var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.
Quoted 9 lines
In this case, his snapshots are all of the root.
I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount
What is the the result of running:
mount | grep /var
Yep.
Signature · 1 line
Chris Murphy
Am 04.09.19 um 21:07 schrieb Chris Murphy:
Quoted 21 lines · Remi Gauvin
Quoted 20 lines · Chris Murphy
Quoted 11 lines
Quoted 2 lines
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or
/var/libwhich means trying to snapshot/var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.In this case, his snapshots are all of the root.
I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount
Quoted 7 lines
Quoted 3 lines
What is the the result of running:
mount | grep /varYep.
Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.
That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).
Hi,
I would agree with Oliver's conclusion:
the relevant directories are all tmpfs mounts.
root@ld5505:~# mount | grep /var /dev/sdbq3 on /var/cache type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=260,subvol=/@cache) /dev/sdbq3 on /var/lib/vz/images type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=262,subvol=/@images) lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) tmpfs on /var/lib/ceph/osd/ceph-122 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-123 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-105 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-92 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-112 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-77 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-76 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-87 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-119 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-79 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-115 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-98 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-100 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-89 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-108 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-120 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-82 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-118 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-104 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-95 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-1 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-96 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-117 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-116 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-99 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-106 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-110 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-97 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-81 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-121 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-88 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-94 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-113 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-107 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-101 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-78 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-93 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-85 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-103 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-102 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-109 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-114 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-80 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-111 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-83 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-86 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-91 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-84 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-90 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-8 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-9 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-10 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-11 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-12 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-13 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-14 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-15 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-16 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-17 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-18 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-19 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-21 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-22 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-23 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-24 type tmpfs (rw,relatime)
Am 04.09.2019 um 22:52 schrieb Oliver Freyermuth:
Quoted 30 lines
Am 04.09.19 um 21:07 schrieb Chris Murphy:
Quoted 13 lines · Remi Gauvin
Quoted 12 lines · Chris Murphy
Quoted 9 lines
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or
/var/libwhich means trying to snapshot/var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.In this case, his snapshots are all of the root.
Quoted 10 lines
Quoted 7 lines
I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount
What is the the result of running:
mount | grep /varYep.
Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).
Dear Thomas,
Am 05.09.19 um 10:50 schrieb Thomas Schneider:
Quoted 4 lines
Hi,
I would agree with Oliver's conclusion:
the relevant directories are all tmpfs mounts.
this of course easily explains it and btrfs can't do anything about these when snapshotting.
In case you want to debug the issue in more detail (e.g. how to extract this information stored in tmpfs from LVM for example), we can of course take this to the ceph-users list (or contact me directly).
Cheers,
Oliver
Quoted 106 lines
root@ld5505:~# mount | grep /var /dev/sdbq3 on /var/cache type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=260,subvol=/@cache) /dev/sdbq3 on /var/lib/vz/images type btrfs (rw,noatime,compress=lzo,ssd,space_cache=v2,subvolid=262,subvol=/@images) lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) tmpfs on /var/lib/ceph/osd/ceph-122 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-123 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-105 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-92 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-112 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-77 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-76 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-87 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-119 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-79 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-115 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-98 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-100 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-89 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-108 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-120 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-82 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-118 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-104 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-95 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-1 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-96 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-117 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-116 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-99 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-106 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-110 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-97 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-81 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-121 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-88 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-94 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-113 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-107 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-101 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-78 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-93 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-85 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-103 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-102 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-109 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-114 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-80 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-111 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-83 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-86 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-91 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-84 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-90 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-8 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-9 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-10 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-11 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-12 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-13 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-14 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-15 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-16 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-17 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-18 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-19 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-21 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-22 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-23 type tmpfs (rw,relatime) tmpfs on /var/lib/ceph/osd/ceph-24 type tmpfs (rw,relatime)Am 04.09.2019 um 22:52 schrieb Oliver Freyermuth:
Quoted 30 lines
Am 04.09.19 um 21:07 schrieb Chris Murphy:
Quoted 13 lines · Remi Gauvin
Quoted 12 lines · Chris Murphy
Quoted 9 lines
I don't really know how snapper works.
The way 'btrfs subvolume snapshot' works, you must point it to a subvolume. It won't snapshot a regular directory and from what you posted above, there are no subvolumes in /var or
/var/libwhich means trying to snapshot /var/lib/ceph/osd/ceph-.... would fail. So maybe it's failing but snapper doesn't show the error. I'm not really sure.In this case, his snapshots are all of the root.
Quoted 10 lines
Quoted 7 lines
I don't know how Ceph works, but since we already confirmed that there are no subvolumes under /var, the only other explanation is that /var/lib/ceph/osd/ceph-<n> is a submount
What is the the result of running:
mount | grep /varYep.
Looking at Thomas' mail on the Ceph-users list:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036679.html
I deduce he is using Ceph with the Bluestore backend, which indeed means that /var/lib/ceph/osd/ceph.<n> are tmpfs mounts which are completeley ephemeral and are created from LVM metadata of the actual Ceph OSD disks.That would of course also explain why they are not part of any btrfs snapshot of / (and that also means there is no need to backup anything).
Can you share any update on this issue?
-------------------------------
Hi,
I was thinking of this, too. But it does not apply.
root@ld5507:~# btrfs su list -to /var/lib ID gen top level path -- --- --------- ---- root@ld5507:~# btrfs su list -to /var ID gen top level path -- --- --------- ----
And there are files in other directories:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mgr/ceph-ld5507/ insgesamt 4 -rw-r--r-- 1 ceph ceph 61 Mai 28 14:33 keyring
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/mon/ceph-ld5507/ insgesamt 12 -rw------- 1 ceph ceph 77 Mai 28 14:33 keyring -rw-r--r-- 1 ceph ceph 8 Mai 28 14:33 kv_backend -rw-r--r-- 1 ceph ceph 3 Aug 23 09:41 min_mon_release drwxr-xr-x 1 ceph ceph 244 Aug 26 18:37 store.db
Only this directories /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-<id>/ are empty:
root@ld5507:~# ls -l /.snapshots/158/snapshot/var/lib/ceph/osd/ceph-219/ insgesamt 0
To create a snapshot I run this command:
snapper create --type single --description "validate /var/lib/ceph/osd/ceph-<n>"
Am 28.08.2019 um 00:24 schrieb Chris Murphy:
Quoted 19 lines · Thomas Schneider
Quoted 7 lines
However, I run into an issue and need to restore various files.
I thought that I could simply take the files from a snapshot created before. However, the files required don't exist in any snapshot!
Therefore I have created a new snapshot manually to verify if the files will be included, but there's nothing.
Snapshots are not recursive on Btrfs. The snapshot will not extend into nested subvolumes. Check to see if you are snapshotting the proper subvolume.
# btrfs sub list -to /var/lib # btrfs sub list -to /var/In some sense these are redundant, I'm not sure if your
/var/libis a subvolume or not. Also please include the exact snapshot command you're making.