Skip to contentSkip to conversation

LVM 0.8final for 2.2.15/2.2.16?

linux-lvm Paul Jakma 6 participants 9 messages Jun 7, 2000 – Nov 27, 2020 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Paul Jakma

author
#1 lore raw
Quoted 2 lines · Jan Niehusmann

The snapshot logic simply doesn't know what a filesystem is. It just clones the block device.

this i knew.. :)

Quoted 4 lines

So the fs may be in en inconsistent state, if you do the snapshot while some write operation is in progress on the cloned filesystem.

ok. so the solution is what? a call in the vfs to complete pending writes, and hold off on modifying any blocks?

(i take it the vfs caches writes somehow before informing LVM of the dirty blocks, or not?)

Quoted 3 lines

Note that cloning a mounted fs usually works, as well as mounting a unclean filesystem without fsck usually works - but it's not reliably.

how unreliable? how busy does the fs have to be?

anyway, for my purposes snapshot does the job. the choice is:

a) backup of a snapshot - inconsistencies hopefully limited to 10**1 order of blocks.

b) backup of a live fs, which is so slow that data inconsistency might be in the order of 10**3 blocks or more.

also, the fs's that are at risk - heavily written to - are usually things like mail and news queues, ie scratch space. These i can afford not to backup. :)

Quoted 4 lines

Jan

regards,

Signature · 5 lines
Paul Jakma	[email protected]
PGP5 key: http://www.clubi.ie/jakma/publickey.txt
-------------------------------------------
Fortune:
An engineer is someone who does list processing in FORTRAN.
#2 lore raw
Quoted 6 lines · Jan Niehusmann
Quoted 3 lines

The snapshot logic simply doesn't know what a filesystem is. It just clones the block device.

Quoted 8 lines

this i knew.. :)

Quoted 3 lines

So the fs may be in en inconsistent state, if you do the snapshot while some write operation is in progress on the cloned filesystem.

ok. so the solution is what? a call in the vfs to complete pending writes, and hold off on modifying any blocks?

IMHO when creating a snapshot LVM could simply sync all outstanding buffers for the block device vialog block_fsync() (not sure if this is already done -- Heinz care to comment?).
This should also take care of pending fs meta information thus we should get a consistent state.

The question is how atomic this is? Since fsync_dev() currently does a lock_kernel() is might even be sufficient.

Any reason why this is not suffiecient?

Michael

--

Michael Marxmeier           Marxmeier Software AG
E-Mail: [email protected]      Besenbruchstrasse 9
Phone : +49 202 2431440     42285 Wuppertal, Germany
Fax   : +49 202 2431420     http://www.msede.com/
author
#3 lore raw
Quoted 2 lines · Michael Marxmeier

IMHO when creating a snapshot LVM could simply sync all outstanding buffers for the block device vialog block_fsync() (not sure if this

to be 100% safe there must be no possibility that some fs code could run between block_fsync() and the actual point of snapshot creation i think. (right?)

Quoted 4 lines

does a lock_kernel() is might even be sufficient.

Any reason why this is not suffiecient?

if you can be sure that lvm-snapshot won't be intterrupted between the sync and the actual snapshot, then it should be ok, shouldn't it?

Quoted 2 lines

Michael

regards,

Signature · 5 lines
Paul Jakma	[email protected]
PGP5 key: http://www.clubi.ie/jakma/publickey.txt
-------------------------------------------
Fortune:
The unfacts, did we have them, are too imprecisely few to warrant our certitude.
#4 lore raw

I have followed only part of this thread, but the gest I get is that people want to take an LVM snapshot of a file system, and the issue at hand is the status of the file system after the sync. I would like to make some remarks based on my experience with other volume managers and file systems. If all or most of this is already a piece of cake for you, please ignore it, but I reckon that there will be people on the list (or reading the archives) that will find this useful.

1) To be useful the snapshot must be "atomic", which means that the snapshotted LV contains an image which conforms to the orginal at a certain time. Since creating the snapshot usually involves some copying of data blocks (to put it mildly) during which you do not pause the entire system, a smart mechanism must be created to maintain this "illusion" of atomicity.

In HP's LVM a snapshot can only be created by splitting of a mirror copy from a mirrored LV (thus decreasing the number of mirror copies of the volume. It is this reason by 3-way mirroring is supported by HP LVM). To create a snapshot one usually first extends the number of mirror copies and then splits off the freshly created mirror.

The Veritas eXtended File System (vxfs) has a built-in snapshot system which works kind-a interesting. Instead of doing a full block device copy of the file system, it uses an "overflow" block device where it saves the originals of a changed block in the original block device. When looking at the snapshot, the vxfs first checks the overflow area if a copy of the requested block is available there. If it is, that block is returned, if it isn't, the block is read from the underlying original since it obviously hasn't been changed since the creation of the snapshot (otherwise the original would have been present in the overflow area). In the worst case the overflow area must be as big as the original, but in typical cases it needs only be 10% of the size of the original. After system reboot, the snapshot copy is gone.

I would guess that such a volatile snapshot facility could be made into a generic feature available for every block device!

2) If you have a snapshot of a logical volume, the file system in there is always corrupt and needs to be fsck'ed. The point in time (atomic) creating of the snapshot resembles a system crash as far as the content of the snapshot is concerned. An fsck is therefore necessary. (A nice feature of the vxfs snapshot is that this fsck is not necessary, because the feature is implemented at the *file system* level).

3) People have been searching for a long time for a method to prevent this fsck. You would need to have full cooperatioon with the file system code for this. The fs should support a "quiesce" function (through the vfs layer) which would result in a complete update of all ondisk data of the fs. A complete block sync is not enough because an fs might have incore data that should be flushed but which is not in the block buffer cache (think:
inode cache, log, B-tree info). Doing a full sync just before the atomic snapshot is a good idea however because it limits the damage fsck must repair.

And, but: READ ON:

4) Even if we could quiesce the fs, the resulting snapshot would still be partially corrupt because of the fact that we have (could have) open files in the file system. If an application updates its data with more than one write() system call, and the snapshot creation happens between two consecutive write()'s, the applications ondisk data is corrupt (from an application point of view). What we normally do in complex backup situations is stop the application, sync the fs, create the snapshot, start the application, backup the snapshot. In that scenario we have a stable copy of the application's data with only a minimal application downtime. This scenario also applies if you use hardware RAID snapshot features such as the Business Continuity Volumes of EMC's Symmetrix, or the Business Copy feature of HP's XP256.

5) So, ideally we would need an "application quiesce" in which we can instruct the application to update its ondisk image by making all necessary changes to its disk data (flush()ing) and informing the operating system of its quiesced state, upon which the OS could make the snapshot and free the application to make changes again. Unix just does not support this particular model of application/OS interaction. And, most applications are internally not architected to easily support a quiesce. And the ones that are, are usually database management systems (such as Oracle) for which you can buy online backup features (such as Oracle Enterprise Backup Utility) with which you can create a stable copy of the database without snapshots or other features.

++Jos

And thus it came to pass that Paul Jakma wrote:
(on Thu, Jun 08, 2000 at 01:47:35AM +0100 to be exact)

Quoted 27 lines · Michael Marxmeier
Quoted 3 lines

IMHO when creating a snapshot LVM could simply sync all outstanding buffers for the block device vialog block_fsync() (not sure if this

to be 100% safe there must be no possibility that some fs code could run between block_fsync() and the actual point of snapshot creation i think. (right?)

Quoted 4 lines

does a lock_kernel() is might even be sufficient.

Any reason why this is not suffiecient?

if you can be sure that lvm-snapshot won't be intterrupted between the sync and the actual snapshot, then it should be ok, shouldn't it?

Quoted 2 lines

Michael

regards,

Signature · 5 lines
Paul Jakma	[email protected]
PGP5 key: http://www.clubi.ie/jakma/publickey.txt
-------------------------------------------
Fortune:
The unfacts, did we have them, are too imprecisely few to warrant our certitude.
Signature · 2 lines
The InSANE quiz master is always right!
(or was it the other way round? :-)
#5 lore raw
Quoted 8 lines · Jos Visser

I have followed only part of this thread, but the gest I get is that people want to take an LVM snapshot of a file system, and the issue at hand is the status of the file system after the sync. I would like to make some remarks based on my experience with other volume managers and file systems. If all or most of this is already a piece of cake for you, please ignore it, but I reckon that there will be people on the list (or reading the archives) that will find this useful.

Quoted 6 lines

1) To be useful the snapshot must be "atomic", which means that the snapshotted LV contains an image which conforms to the orginal at a certain time. Since creating the snapshot usually involves some copying of data blocks (to put it mildly) during which you do not pause the entire system, a smart mechanism must be created to maintain this "illusion" of atomicity.

The problem is that it is not enough to sync the file systems to get an atomic copy; you must sync all applications too. For example consider a database that uses user space journaling using fsync:
to make its disk files consistent after the snapshot it requires both the log write and the data write. When one is missing the log needs to be replayed, which requires writes.

Quoted 6 lines

The Veritas eXtended File System (vxfs) has a built-in snapshot system which works kind-a interesting. Instead of doing a full block device copy of the file system, it uses an "overflow" block device where it saves the originals of a changed block in the original block device. When looking at the snapshot, the vxfs first checks the overflow area if a copy of the requested

Quoted 8 lines

block is available there. If it is, that block is returned, if it isn't, the block is read from the underlying original since it obviously hasn't been changed since the creation of the snapshot (otherwise the original would have been present in the overflow area). In the worst case the overflow area must be as big as the original, but in typical cases it needs only be 10% of the size of the original. After system reboot, the snapshot copy is gone.

This is basically how the current LVM snapshot mechanism works. The overflow blocks are configurable. The snapshots are gone after a reboot.

Quoted 3 lines

I would guess that such a volatile snapshot facility could be made into a generic feature available for every block device!

Yes, it's called LVM @)

Quoted 7 lines

5) So, ideally we would need an "application quiesce" in which we can instruct the application to update its ondisk image by making all necessary changes to its disk data (flush()ing) and informing the operating system of its quiesced state, upon which the OS could make the snapshot and free the application to make changes again. Unix just does not support this particular model of application/OS interaction. And,

There is an easy workaround: make the snapshots writeable and allow an application ``fsck'' or log replay in the snapshot (if it can't handle that it is not crash safe anyways)

-Andi

#6 lore raw
Quoted 4 lines · Andi Kleen

consider a database that uses user space journaling using fsync:
to make its disk files consistent after the snapshot it requires both the log write and the data write. When one is missing the log needs to be replayed, which requires writes.

but doesn't the call to block_fsync that Heinz confirmed exists cover this?

nothing can cover the case where app data consistency depends on a future write(). But that's the app's problem, and anyway a good database should be consistent/recover itself if it's died between write(?)'s. right?

(in which case lvm snapshot is perfectly suitable for backing up databases..)

--paulj

#7 lore raw

And thus it came to pass that Paul Jakma wrote:
(on Thu, Jun 08, 2000 at 12:49:53PM +0100 to be exact)

Quoted 10 lines · Andi Kleen
Quoted 5 lines

consider a database that uses user space journaling using fsync:
to make its disk files consistent after the snapshot it requires both the log write and the data write. When one is missing the log needs to be replayed, which requires writes.

but doesn't the call to block_fsync that Heinz confirmed exists cover this?

Quoted 6 lines

nothing can cover the case where app data consistency depends on a future write(). But that's the app's problem, and anyway a good database should be consistent/recover itself if it's died between write(?)'s. right?

(in which case lvm snapshot is perfectly suitable for backing up databases..)

Most databases want you to store their log and data spaces in different (logical) volumes. To create a consistent image you would want to be able to snapshot multiple logical volumes atomically in one operation. HP's LVM supports this through the "multiple atomical lvsplit" feature (as of HP-UX 10 if memory serves me right).

++Jos

Signature · 2 lines
The InSANE quiz master is always right!
(or was it the other way round? :-)
#8 lore raw
Quoted 17 lines · Jos Visser

And thus it came to pass that Paul Jakma wrote:
(on Thu, Jun 08, 2000 at 12:49:53PM +0100 to be exact)

Quoted 14 lines · Andi Kleen
Quoted 5 lines

consider a database that uses user space journaling using fsync:
to make its disk files consistent after the snapshot it requires both the log write and the data write. When one is missing the log needs to be replayed, which requires writes.

but doesn't the call to block_fsync that Heinz confirmed exists cover this?

nothing can cover the case where app data consistency depends on a future write(). But that's the app's problem, and anyway a good database should be consistent/recover itself if it's died between write(?)'s. right?

Quoted 5 lines
Quoted 2 lines

(in which case lvm snapshot is perfectly suitable for backing up databases..)

Most databases want you to store their log and data spaces in different (logical) volumes.

Exactly.

That's why i'll implement atomic activation off multiple snapshot.

The idea is:

- prepare multiple _inactive_ snapshots
- activate them afterwards at once

Heinz

Quoted 10 lines

To create a consistent image you would want to be
able to snapshot multiple logical volumes atomically in one operation. HP's LVM supports this through the "multiple atomical lvsplit" feature (as of HP-UX 10 if memory serves me right).

++Jos

Signature · 2 lines
The InSANE quiz master is always right!
(or was it the other way round? :-)
Signature · 9 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                Sistina Software Inc.
Senior Consultant/Developer                      Bartningstr. 12
                                                 64289 Darmstadt
                                                 Germany
[email protected]                           +49 6151 710386
                                                      FAX 710396
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#9 lore raw
In reply to Michael Marxmeier
Quoted 12 lines · Michael Marxmeier
Quoted 12 lines · Jan Niehusmann
Quoted 3 lines

The snapshot logic simply doesn't know what a filesystem is. It just clones the block device.

this i knew.. :)

Quoted 3 lines

So the fs may be in en inconsistent state, if you do the snapshot while some write operation is in progress on the cloned filesystem.

Quoted 6 lines
Quoted 2 lines

ok. so the solution is what? a call in the vfs to complete pending writes, and hold off on modifying any blocks?

IMHO when creating a snapshot LVM could simply sync all outstanding buffers for the block device vialog block_fsync() (not sure if this is already done -- Heinz care to comment?).

Yes, in case a new snapshot is created the buffers of the original LV are flushed using fsync_dev(). Please have a look at the end of lvm_do_lv_create() in lvm.c.

Quoted 5 lines

This should also take care of pending fs meta information thus we should get a consistent state.

The question is how atomic this is? Since fsync_dev() currently does a lock_kernel() is might even be sufficient.

It should do the job (see comment bellow).

Quoted 3 lines

Any reason why this is not suffiecient?

The only case could be any kind of "private" metadata a filesystem keeps outside the cache.
Any comments on that?

Heinz

Signature · 9 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                Sistina Software Inc.
Senior Consultant/Developer                      Bartningstr. 12
                                                 64289 Darmstadt
                                                 Germany
[email protected]                           +49 6151 710386
                                                      FAX 710396
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-