File changing in snapshot
I seem to have observed a file on a (writable) snapshot changing although there were no writes occuring on the snapshot itself. This is not supposed to happen, right?
Sequence of events:
1. A (writable) snapshot @home-2014-04-16 is taken on a @home subvolume mounted at /home.
2. The current subvolume (@home) is used (via /home) for continued write access, no writes are supposed to occur on the snapshot.
3. The snapshot @home-2014-04-16 is backed up (using rdiff-backup) to a different disk.
4. As the backup is compared to the snapshot @home-2014-04-16, one file differs.
Further analysis:
The file in @home-2014-04-16 changed its contents (but not mtime or other metadata) after being backup up from that snapshot. 5 bytes differ.
The previous snapshot @home-2014-04-10 contains the file in its original form, which is identical to the backup just taken from @home-2014-04-16.
The file change appears identically
- on the last snapshot @home-2014-04-16 and
- on the current subvolume used for writing (@home).
btrfs scrub reports 0 errors.
At the time the snapshot @home-2014-04-16 was taken, the file (an Excel file) was probably accessed through Samba, which supposedly uses kernel oplocks (if that makes a difference).
Versions:
- Kernel 3.11.0-19-generic (Ubuntu 13.10)
- btrfs-tools 0.19+20130705-1
- Samba 3.6.18
Any ideas?
Quoted 32 lines · Oliver O.
I seem to have observed a file on a (writable) snapshot changing although there were no writes occuring on the snapshot itself. This is not supposed to happen, right?
Sequence of events:
1. A (writable) snapshot @home-2014-04-16 is taken on a @home subvolume mounted at /home.
2. The current subvolume (@home) is used (via /home) for continued write access, no writes are supposed to occur on the snapshot.
3. The snapshot @home-2014-04-16 is backed up (using rdiff-backup) to a different disk.
4. As the backup is compared to the snapshot @home-2014-04-16, one file differs.
Further analysis:
The file in @home-2014-04-16 changed its contents (but not mtime or other metadata) after being backup up from that snapshot. 5 bytes differ.
The previous snapshot @home-2014-04-10 contains the file in its original form, which is identical to the backup just taken from @home-2014-04-16.
The file change appears identically
- on the last snapshot @home-2014-04-16 and
- on the current subvolume used for writing (@home).btrfs scrub reports 0 errors.
Quoted 8 lines
At the time the snapshot @home-2014-04-16 was taken, the file (an Excel file) was probably accessed through Samba, which supposedly uses kernel oplocks (if that makes a difference).
Versions:
- Kernel 3.11.0-19-generic (Ubuntu 13.10)
- btrfs-tools 0.19+20130705-1
- Samba 3.6.18
Was this a nodatacow file?
-chris
Am 17.04.2014 17:56, schrieb Chris Mason:
Quoted 8 lines · Oliver O.
Quoted 3 lines
I seem to have observed a file on a (writable) snapshot changing although there were no writes occuring on the snapshot itself. This is not supposed to happen, right?
Was this a nodatacow file?
-chris
No. Mount options:
/dev/sda2 on /home type btrfs (rw,noatime,nodiratime,subvol=@home)
Am 17.04.2014 18:11, schrieb Oliver O.:
Quoted 7 lines
Am 17.04.2014 17:56, schrieb Chris Mason:
Quoted 5 lines · Oliver O.
Quoted 3 lines
I seem to have observed a file on a (writable) snapshot changing although there were no writes occuring on the snapshot itself. This is not supposed to happen, right?
Quoted 8 lines
Quoted 3 lines
Was this a nodatacow file?
-chris
No. Mount options:
/dev/sda2on /home type btrfs (rw,noatime,nodiratime,subvol=@home)
Some additional observations:
- Subvolume used for writing: @home
- Snapshot used for backup: @home-2014-04-16
- Preceding snapshot: @home-2014-04-10
# lsattr -v @home-2014-04-10/path/to/MyFile.xls @home-2014-04-16/path/to/MyFile.xls @home/path/to/MyFile.xls 36066 ---------------- @home-2014-04-10/path/to/MyFile.xls 36066 ---------------- @home-2014-04-16/path/to/MyFile.xls 36066 ---------------- @home/path/to/MyFile.xls
# btrfs subvolume find-new @home-2014-04-10 99999999 transid marker was 180026
# btrfs subvolume find-new @home-2014-04-16 99999999 transid marker was 194551
# btrfs subvolume find-new @home 99999999 transid marker was 197735
# btrfs subvolume find-new @home 36066 | grep "MyFile.xls" inode 24426 file offset 0 len 20480 disk start 9108635648 offset 0 gen 193090 flags NONE path/to/MyFile.xls inode 24426 file offset 20480 len 36864 disk start 25510342656 offset 20480 gen 36067 flags NONE path/to/MyFile.xls inode 24426 file offset 57344 len 4096 disk start 9108582400 offset 0 gen 193090 flags NONE path/to/MyFile.xls
Conclusions:
The sequence of events seems to be:
1. The file was changed with generation 193090.
2. The snapshot for backup (@home-2014-04-16) was taken (generation 194551).
3. As the backup was reading from the snapshot, it was seeing stale data (MyFile still at generation 36067).
4. As the backup comparison was reading from the snapshot, is was seeing more recent data, causing a discrepancy with the stale data just backed up.
It seems that the snapshot needed some time to become stable for reading. From a file system user's point of view, I'd expect a snapshot to be atomic and stable at any time. Maybe some kind of caching problem here?
Maybe it's time to file a bug report, but first: suggestions and ideas appreciated.
Solved, false alarm - Re: File changing in snapshot
Am 17.04.2014 23:29, schrieb Oliver O.:
Quoted 7 lines
Conclusions:
The sequence of events seems to be:
1. The file was changed with generation 193090.
2. The snapshot for backup (@home-2014-04-16) was taken (generation 194551).
3. As the backup was reading from the snapshot, it was seeing stale data (MyFile still at generation 36067).
This conclusion was wrong:
Rdiff-backup never really considered the modified Excel file for backup, as its file modification time did not indicate that the file was newer than the last backup. So rdiff-backup always took the file copy already available in the previous backup.
What breaks the incremental backup mechanism is Excel modifying a file, then changing its modification timestamp back to its original value (see comment 10 at https://bugzilla.samba.org/show_bug.cgi?id=1601#c10).
Quoted 1 line
4. As the backup comparison was reading from the snapshot, is was seeing more recent data, causing a discrepancy with the stale data just backed up.
The discrepancy between the file's modified contents were discovered because the comparison was content-based (MD5 checksums) and compared every file regardless of modification dates.
Quoted 5 lines
It seems that the snapshot needed some time to become stable for reading. From a file system user's point of view, I'd expect a snapshot to be atomic and stable at any time. Maybe some kind of caching problem here?
Maybe it's time to file a bug report, but first: suggestions and ideas appreciated.
Result: False alarm - this is definitely not a Btrfs problem. In this case snapshot behavior was as expected: atomic and stable.
Sorry for any irritation.
Re: Solved, false alarm - Re: File changing in snapshot
Quoted 7 lines · Oliver O.
Am 17.04.2014 23:29, schrieb Oliver O.:
Result: False alarm - this is definitely not a Btrfs problem. In this case snapshot behavior was as expected: atomic and stable.
Sorry for any irritation.
But, mystery solved and documented for later generations ;) Thanks for nailing it down and posting the result.
-chris