Skip to contentSkip to conversation

fsync() and LVM

linux-lvm Marco Colombo 14 participants 39 messages Mar 13, 2009 – Mar 19, 2009

lore
Share & tools
lore
#0 lore raw

Hi, I'm a long time user of both PostgreSQL and LVM. So far I've been quite happy with both. But a recent thread on the PostgreSQL list made me unconfortable. What is this thing they're referring to, fsync()'s being ignored? Makes me feel like I'm running on thin ice, without even knowing. Before I start phasing out LVM from all my PostgreSQL installations (as they suggest), I'd like to hear some kind of confirmation. This is quite scary.

http://archives.postgresql.org/pgsql-general/2009-03/msg00204.php

In my understanding:

fsync(): force data from OS memory to disk (ending up in the disk cache) write barrier: force data from disk cache to disk platters

If you disable write-back cache on the disks, you no longer need write barriers. But apparently they claim LVM being unsafe even with disks caches in write-thru mode, which surprises me a lot.

.TM.

#1 lore raw
Quoted 8 lines · Marco Colombo

Hi, I'm a long time user of both PostgreSQL and LVM. So far I've been quite happy with both. But a recent thread on the PostgreSQL list made me unconfortable. What is this thing they're referring to, fsync()'s being ignored? Makes me feel like I'm running on thin ice, without even knowing. Before I start phasing out LVM from all my PostgreSQL installations (as they suggest), I'd like to hear some kind of confirmation.

http://archives.postgresql.org/pgsql-general/2009-03/msg00204.php

The discussion doesn't make a lot of sense. fsync() is a filesystem call - it can't possibly be handled (or ignored) at a lower level because the lowel level doesn't know which blocks belong to the file.
I *can* imagine that perhaps the raw block writes used by the filesystem code might be ignored - or improperly cached. Clearly, they are not ignored (filesystems do get updated) - so if there is any substance to the charge, it must be that LVM reorders writes somehow. Caching doesn't really break anything - it is the *reordering* of writes that could be a problem. A "write barrier" says "finish these writes before you start any more, but otherwise reorder how you like". I did some experiments with iostat, and I am convinced that LVM does not itself do any reordering of writes.

Here is my theory as to what is really going on:

LVM is not really "ignoring" fsync(), because it would never see it. However, in the presence of hardware writeback caching in disk drives, fsync() would need to tell the hardware to "finish all these writes before you start any more" (i.e. - a write barrier) for fsync() to be effective.

I suspect that LVM simply fails to pass the *write barrier* through to underlying layers (i.e. ignores the writebarrier call). Thus, you should be fine if you simply turn off writeback caching in your disk drives. If you could guarrantee that disk drives would be powered long enough after the main system stops - even a drive write back cache would not be much of a risk.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#2 lore raw
Quoted 5 lines · Stuart D. Gathman

I suspect that LVM simply fails to pass the *write barrier* through to underlying layers (i.e. ignores the writebarrier call). Thus, you should be fine if you simply turn off writeback caching in your disk drives. If you could guarrantee that disk drives would be powered long enough after the main system stops - even a drive write back cache would not be much of a risk.

The big question in my mind is which software layers don't get to see the write barrier. If any of them can reorder writes, that could (however unlikely) lead to data corruption.

#3 lore raw
In reply to Stuart D. Gathman

Let's try to clear up the confusion.

Kernel device-mapper (which lvm uses) does not support write barriers except in very restricted circumstances (when only one device is involved and the mapping is trivial). If dm receives a write barrier which is not supported it notifies the caller (typically a filesystem) so appropriate action can be taken if it wishes.

Several kernels releases ago, the implementation of the 'flush device' operation in the block layer was changed from a simple function call that dm supported to a mechanism involving barriers that is trickier for dm to support. Previously 'flush' could not fail and so callers do not generally have strategies to handle such a situation.

The latest of several attempts to support barriers is contained in patches here:
http://patchwork.kernel.org/project/dm-devel/list/?q=barriers

Please review and test if you are interested!

Alasdair

Signature · 1 line
[email protected]
#4 lore raw
Quoted 5 lines · Alasdair G Kergon

Several kernels releases ago, the implementation of the 'flush device' operation in the block layer was changed from a simple function call that dm supported to a mechanism involving barriers that is trickier for dm to support. Previously 'flush' could not fail and so callers do not generally have strategies to handle such a situation.

The 'caller' here would be fsync() in the FS. What strategies are available to handle a failing 'flush'? It there anything that can be done at application level (userland)?

More than anything, does LVM (or device mapper) really reorder writes? Is it safe with disk caches in write-thru mode? (hdparm -W0)

.TM.

#5 lore raw
In reply to Alasdair G Kergon
Quoted 7 lines

Let's try to clear up the confusion.

Kernel device-mapper (which lvm uses) does not support write barriers except in very restricted circumstances (when only one device is involved and the mapping is trivial). If dm receives a write barrier which is not supported it notifies the caller (typically a filesystem) so appropriate action can be taken if it wishes.

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

- Dietmar

#6 lore raw
Quoted 10 lines · Dietmar Maurer
Quoted 7 lines

Let's try to clear up the confusion.

Kernel device-mapper (which lvm uses) does not support write barriers except in very restricted circumstances (when only one device is involved and the mapping is trivial). If dm receives a write barrier which is not supported it notifies the caller (typically a filesystem) so appropriate action can be taken if it wishes.

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches. Note that with multiple devices, even a FIFO write cache could cause reordering between devices (one device could finish faster than another).

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#7 lore raw
Quoted 5 lines · Stuart D. Gathman

On Sat, 14 Mar 2009, Dietmar Maurer wrote:
It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches. Note that with multiple devices, even a FIFO write cache could cause reordering between devices (one device could finish faster than another).

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(), that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true, you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

.TM.

#8 lore raw
Quoted 7 lines · Marco Colombo
Quoted 5 lines · Stuart D. Gathman

On Sat, 14 Mar 2009, Dietmar Maurer wrote:
It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches. Note that with multiple devices, even a FIFO write cache could cause reordering between devices (one device could finish faster than another).

Quoted 6 lines

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(), that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true, you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

Seeing as my /var (with both postfix & courier-imap using it for mail storage) has been on lvm for almost 4 years, that would be news to me...

;)

Signature · 3 lines
Best regards,

Charles
#9 lore raw

2009/3/16, Charles Marcus <[email protected]>:

Quoted 2 lines

Seeing as my /var (with both postfix & courier-imap using it for mail
storage) has been on lvm for almost 4 years, that would be news to me...

And how often has the computer crashed needing an fsck in those years? It's most likely no problem if the fs is always unmounted cleanly.

Best
Martin

#10 lore raw
Quoted 2 lines · Martin Schr�der

And how often has the computer crashed needing an fsck in those years? It's most likely no problem if the fs is always unmounted cleanly.

There have been probably 4 unclean shutdowns (due to extended power outages) in these 4 years, 2 of which required an extended fsck...

Running reiserfs too...

Zero problems to date (knock on wood)...

Signature · 3 lines
Best regards,

Charles
#11 lore raw
Quoted 11 lines · Martin Schr�der
Quoted 1 line

And how often has the computer crashed needing an fsck in those

years?

Quoted 1 line

It's most likely no problem if the fs is always unmounted cleanly.

There have been probably 4 unclean shutdowns (due to extended power outages) in these 4 years, 2 of which required an extended fsck...

Running reiserfs too...

Zero problems to date (knock on wood)...

The question is if fsync is implemented correctly or not?

- Dietmar

#12 lore raw
In reply to Charles Marcus
Quoted 15 lines · Charles Marcus
Quoted 7 lines · Marco Colombo
Quoted 5 lines · Stuart D. Gathman

On Sat, 14 Mar 2009, Dietmar Maurer wrote:
It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches. Note that with multiple devices, even a FIFO write cache could cause reordering between devices (one device could finish faster than another).

Quoted 6 lines

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(), that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true, you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

Quoted 5 lines

Seeing as my /var (with both postfix & courier-imap using it for mail storage) has been on lvm for almost 4 years, that would be news to me...

;)

Believe me or not, they both depend on fsync(). Anyway, even if you lost a message, how do you expect to know? If you have any user base large enough, you're used to 'missing' messages (99% of the user-deleted or user-never-sent kind). A truly lost one may have gone missed in the noise.

A lying fsync() doesn't blow all your mail repository up, just you may loose one/two messages on a crash. Or a transaction, speaking of databases. If that's the case, I would like to know, that's all.

.TM.

#13 lore raw
Quoted 2 lines · Marco Colombo
Quoted 2 lines
Quoted 2 lines

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(), that data doesn't even get to the controller, and it doesn't matter

If that was the case, then just *attempting* to call fsync would corrupt your filesystem/database- "dirty" blocks would not actually get written, but still get marked "clean". Clearly, LVM does not interfere with writing to the disk. It is only write barriers (waiting for the writes to actually finish) that don't get passed through (in all but the most simple cases). It simply returns you to the old days when the man page for sync() said "this queues dirty blocks for writing but does not wait for them to finish" and shutdown scripts called sync() multiple times with sleeps in between.

Quoted 3 lines

A lying fsync() doesn't blow all your mail repository up, just you may loose one/two messages on a crash. Or a transaction, speaking of databases. If that's the case, I would like to know, that's all.

Since the fsync() returns "fail" when LVM can't map it to multiple devices, it isn't exactly "lying". And one possible response to a failure might be to wait a bit.

According to the redhat guy, this problem came up when the simple block device "flush" call was replaced with the more complex write barrier. LVM had no problem passing through a simple block device flush. (Why couldn't the simple "flush" call still be available?) I would like to know which kernel version made this change.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#14 lore raw
In reply to Marco Colombo
Quoted 8 lines · Marco Colombo
Quoted 5 lines · Stuart D. Gathman

On Sat, 14 Mar 2009, Dietmar Maurer wrote:
It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches. Note that with multiple devices, even a FIFO write cache could cause reordering between devices (one device could finish faster than another).

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(),

That is clearly wrong - since fsync() isn't LVM's responsibility. I think they mean that fsync() can't garrantee that any writes are actually on the platter.

Quoted 3 lines

that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true,

That is clearly wrong. If writes don't work, nothing works.

Quoted 2 lines

you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers, and then waits for the physical writes to complete. It is only the waiting part that is broken.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#15 lore raw
Quoted 11 lines · Stuart D. Gathman
Quoted 2 lines

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(),

That is clearly wrong - since fsync() isn't LVM's responsibility. I think they mean that fsync() can't garrantee that any writes are actually on the platter.

Quoted 3 lines

that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true,

Quoted 8 lines

That is clearly wrong. If writes don't work, nothing works.

Quoted 2 lines

you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers, and then waits for the physical writes to complete. It is only the waiting part that is broken.

It's a yes or no question... Fsync() either guarantees that the write is committed to physical media so the application can continue knowing that it's own transactional expectations are met (i.e. you can crash and recover that piece of data), or it is broken. If it doesn't wait for completion, it can't possibly report the correct status.

Signature · 2 lines
  Les Mikesell
   [email protected]
#16 lore raw
Quoted 22 lines · Les Mikesell
Quoted 12 lines · Stuart D. Gathman
Quoted 2 lines

No, it's more than that. PostgreSQL gurus say LVM doesn't honor fsync(),

That is clearly wrong - since fsync() isn't LVM's responsibility. I think they mean that fsync() can't garrantee that any writes are actually on the platter.

Quoted 4 lines

that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed
caches. Please read the thread I linked. If what they say it's true,

Quoted 8 lines

That is clearly wrong. �If writes don't work, nothing works.

Quoted 2 lines

you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers, and then waits for the physical writes to complete. �It is only the waiting part that is broken.

Quoted 6 lines

It's a yes or no question... �Fsync() either guarantees that the write is committed to physical media so the application can continue knowing that it's own transactional expectations are met (i.e. you can crash and recover that piece of data), or it is broken. �If it doesn't wait for completion, it can't possibly report the correct status.

This discussion seems a bit bizarre to me. Many apps require data get to stable memory in a well defined way. Barriers is certainly one way to do that, but I don't think barriers are supported by LVM, mdraid, or drbd.

Those are some very significant subsystems. I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#17 lore raw

[linux-lvm] liblvm status question

Is the liblvm project at a state where vgs, lvs or pvs functionality is mostly working? I am writing a program for working with logical volumes and it would be very helpful if I could start integrating liblvm now even if it is still buggy at the moment. Also, where can I find the files or cvs?

Thank you

#18 lore raw

Re: [linux-lvm] liblvm status question

Quoted 6 lines · ben scott

Is the liblvm project at a state where vgs, lvs or pvs functionality is mostly working? I am writing a program for working with logical volumes and it would be very helpful if I could start integrating liblvm now even if it is still buggy at the moment. Also, where can I �find the files or cvs?

Thank you

I think you mean libdevmapper don't you?

Pretty sure libdevmapper is used by the core LVM 2.0 tools. Unfortunately, I'm not aware of any documentation about what the API is. I guess you have to read the source.

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#19 lore raw

Re: [linux-lvm] liblvm status question

Quoted 9 lines · Greg Freemyer
Quoted 6 lines · ben scott

Is the liblvm project at a state where vgs, lvs or pvs functionality is mostly working? I am writing a program for working with logical volumes and it would be very helpful if I could start integrating liblvm now even if it is still buggy at the moment. Also, where can I find the files or cvs?

Thank you

I think you mean libdevmapper don't you?

No, he means liblvm:

http://fedoraproject.org/wiki/Features/liblvm
http://fedoraproject.org/wiki/LVM/liblvm

Patches are just starting to be merged but it's still a work-in-progress:

http://www.redhat.com/archives/lvm-devel/2009-March/msg00008.html

Follow the lvm-devel mailing list to keep track of what's going on.

Regards,
Bryn.

#20 lore raw

Re: [linux-lvm] liblvm status question

Quoted 20 lines · Bryn M. Reeves
Quoted 10 lines · Greg Freemyer
Quoted 7 lines · ben scott

Is the liblvm project at a state where vgs, lvs or pvs functionality is mostly working? I am writing a program for working with logical volumes and it would be very helpful if I could start integrating liblvm now even if it is still buggy at the moment. Also, where can I find the files or cvs?

Thank you

I think you mean libdevmapper don't you?

No, he means liblvm:

http://fedoraproject.org/wiki/Features/liblvm
http://fedoraproject.org/wiki/LVM/liblvm

Patches are just starting to be merged but it's still a work-in-progress:

Quoted 6 lines

http://www.redhat.com/archives/lvm-devel/2009-March/msg00008.html

Follow the lvm-devel mailing list to keep track of what's going on.

Regards,
Bryn.

Thank you.

#21 lore raw

Re: [linux-lvm] liblvm status question

In reply to Bryn M. Reeves
Quoted 13 lines · Bryn M. Reeves
Quoted 9 lines · Greg Freemyer
Quoted 6 lines · ben scott

Is the liblvm project at a state where vgs, lvs or pvs functionality is mostly working? I am writing a program for working with logical volumes and it would be very helpful if I could start integrating liblvm now even if it is still buggy at the moment. Also, where can I �find the files or cvs?

Thank you

I think you mean libdevmapper don't you?

No, he means liblvm:

Quoted 8 lines

http://fedoraproject.org/wiki/Features/liblvm
http://fedoraproject.org/wiki/LVM/liblvm

Patches are just starting to be merged but it's still a work-in-progress:

http://www.redhat.com/archives/lvm-devel/2009-March/msg00008.html

Very cool. I just recommended libdevmapper to a project team doing some work. liblvm looks like a much better fit for them.

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#22 lore raw
In reply to Greg Freemyer
Quoted 6 lines · Greg Freemyer
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers, and then waits for the physical writes to complete. It is only the waiting part that is broken.

Quoted 8 lines
Quoted 6 lines

It's a yes or no question... Fsync() either guarantees that the write is committed to physical media so the application can continue knowing that it's own transactional expectations are met (i.e. you can crash and recover that piece of data), or it is broken. If it doesn't wait for completion, it can't possibly report the correct status.

This discussion seems a bit bizarre to me.

You can't avoid a discussion of expected but missing functionality.

Quoted 8 lines

Many apps require data get
to stable memory in a well defined way. Barriers is certainly one way to do that, but I don't think barriers are supported by LVM, mdraid, or drbd.

Those are some very significant subsystems. I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

If you can't get the completion status from the underlying layer, how can a filesystem possibly implement it?

Quoted 3 lines

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

I though linux ended up doing a sync of the entire outstanding buffered data for a partition with horrible performance, at least on ext3.

Signature · 2 lines
  Les Mikesell
   [email protected]
#23 lore raw
Quoted 28 lines · Les Mikesell
Quoted 18 lines · Greg Freemyer
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

you can't use LVM for anything that needs fsync(), including mail queues
(sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers, and then waits for the physical writes to complete. �It is only the waiting part that is broken.

Quoted 8 lines

It's a yes or no question... �Fsync() either guarantees that the write is committed to physical media so the application can continue knowing that it's own transactional expectations are met (i.e. you can crash and recover
that piece of data), or it is broken. �If it doesn't wait for completion, it
can't possibly report the correct status.

Quoted 1 line

This discussion seems a bit bizarre to me.

You can't avoid a discussion of expected but missing functionality.

Quoted 5 lines

Many apps require data get
to stable memory in a well defined way. �Barriers is certainly one way to do that, but I don't think barriers are supported by LVM, mdraid, or drbd.

Quoted 6 lines
Quoted 3 lines

Those are some very significant subsystems. �I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

If you can't get the completion status from the underlying layer, how can a filesystem possibly implement it?

Barriers is a specific technology and they were just implemented in linux around 2005 I think. (see documentation/barriers.txt)

Surely there was a mechanism in place before that.

Quoted 6 lines
Quoted 3 lines

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

I though linux ended up doing a sync of the entire outstanding buffered data for a partition with horrible performance, at least on ext3.

Yes, I understand fsync is horribly slow in ext3 and that may be the reason. Supposedly much better in ext4. Still if a userspace app calls fsync and in turn the filesystem does something really slow due to the lack of barriers, then this conversation should be about the poor performance of fsync() when using lvm (or mdraid, or drdb), not the total lack of fsync() support.

Quoted 3 lines

--
�Les Mikesell
� [email protected]

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#24 lore raw
Quoted 11 lines · Greg Freemyer
Quoted 6 lines
Quoted 4 lines

Those are some very significant subsystems. I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

If you can't get the completion status from the underlying layer, how can a filesystem possibly implement it?

Barriers is a specific technology and they were just implemented in linux around 2005 I think. (see documentation/barriers.txt)

Surely there was a mechanism in place before that.

I'm not sure that's a reasonable assumption.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

I though linux ended up doing a sync of the entire outstanding buffered data for a partition with horrible performance, at least on ext3.

Quoted 6 lines

Yes, I understand fsync is horribly slow in ext3 and that may be the reason. Supposedly much better in ext4. Still if a userspace app calls fsync and in turn the filesystem does something really slow due to the lack of barriers, then this conversation should be about the poor performance of fsync() when using lvm (or mdraid, or drdb), not the total lack of fsync() support.

I haven't seen anyone claim yet that there is support for fsync(), which must return the status of the completion of the operation to the application. If it does, then the discussion could turn to performance.

Signature · 2 lines
  Les Mikesell
   [email protected]
#25 lore raw
Quoted 32 lines · Les Mikesell
Quoted 5 lines · Greg Freemyer
Quoted 5 lines
Quoted 4 lines

Those are some very significant subsystems. �I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

Quoted 8 lines
Quoted 3 lines

If you can't get the completion status from the underlying layer, how can a
filesystem possibly implement it?

Barriers is a specific technology and they were just implemented in linux around 2005 I think. �(see documentation/barriers.txt)

Surely there was a mechanism in place before that.

I'm not sure that's a reasonable assumption.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

I though linux ended up doing a sync of the entire outstanding buffered data
for a partition with horrible performance, at least on ext3.

Quoted 6 lines

Yes, I understand fsync is horribly slow in ext3 and that may be the reason. �Supposedly much better in ext4. �Still if a userspace app calls fsync and in turn the filesystem does something really slow due to the lack of barriers, then this conversation should be about the poor performance of fsync() when using lvm (or mdraid, or drdb), not the total lack of fsync() support.

Quoted 7 lines

I haven't seen anyone claim yet that there is support for fsync(), which must return the status of the completion of the operation to the application. �If it does, then the discussion could turn to performance.

--
�Les Mikesell
� [email protected]

Is your specific interest to ext3? If so, I suggest you post a question there along the lines of:

Device Mapper does not support barriers if more than one physical device is in use by the LV. If I'm using ext3 on a LV and I call fsync() from user space, how is fsync() implemented. Or is it not?

The ext4 list is <[email protected]>. I see some ext3 stuff posted there, or it may have its own list.

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#26 lore raw
Quoted 5 lines · Greg Freemyer
Quoted 4 lines
Quoted 1 line

I haven't seen anyone claim yet that there is support for fsync(), which

must return the status of the completion of the operation to the application. If it does, then the discussion could turn to performance.

Is your specific interest to ext3?

No, it is whether a useful fsync() is possible over LVM.

Quoted 6 lines

If so, I suggest you post a
question there along the lines of:

Device Mapper does not support barriers if more than one physical device is in use by the LV. If I'm using ext3 on a LV and I call fsync() from user space, how is fsync() implemented. Or is it not?

The point of fsync() is for an application to know that a write has been safely committed, as for example sendmail would do before acknowledging to the sender that a message has been accepted. The question isn't whether an application can call fsync() but rather whether it's return status is lying, making the underlying storage unsuitable for anything that needs reliability.

Signature · 2 lines
  Les Mikesell
   [email protected]
#27 lore raw
Quoted 6 lines · Les Mikesell

The point of fsync() is for an application to know that a write has been safely committed, as for example sendmail would do before acknowledging to the sender that a message has been accepted. The question isn't whether an application can call fsync() but rather whether it's return status is lying, making the underlying storage unsuitable for anything that needs reliability.

Right and for databases this is critical. So enlightenment here would be good.

Sincerely,

Joshua D. Drake

Quoted 1 line
Signature · 4 lines
PostgreSQL - XMPP: [email protected]
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997
#28 lore raw
Quoted 11 lines · Joshua D. Drake
Quoted 6 lines · Les Mikesell

The point of fsync() is for an application to know that a write has been safely committed, as for example sendmail would do before acknowledging to the sender that a message has been accepted. The question isn't whether an application can call fsync() but rather whether it's return status is lying, making the underlying storage unsuitable for anything that needs reliability.

Right and for databases this is critical. So enlightenment here would be good.

Anyone?

Joshua D. Drake

Signature · 4 lines
PostgreSQL - XMPP: [email protected]
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997
#29 lore raw
Quoted 10 lines · Joshua D. Drake
Quoted 9 lines · Joshua D. Drake
Quoted 6 lines · Les Mikesell

The point of fsync() is for an application to know that a write has been safely committed, as for example sendmail would do before acknowledging to the sender that a message has been accepted. The question isn't whether an application can call fsync() but rather whether it's return status is lying, making the underlying storage unsuitable for anything that needs reliability.

Quoted 6 lines
Quoted 2 lines

Right and for databases this is critical. So enlightenment here would be good.

Anyone?

Joshua D. Drake

If a logical volume spans physical devices where write caching is enabled, the results of fsync() can not be trusted. This is an issue with device mapper, lvm is one of a few possible customers of DM.

Now it gets interesting:

Enter virtualization. When you have something like this:

fsync -> guest block device -> block tap driver -> CLVM -> iscsi -> storage -> physical disk.

Even if device mapper passed along the write barrier, would it be reliable? Is every part of that chain going to pass the same along, and how many opportunities for re-ordering are presented in the above?

So, even if its fixed in DM, can fsync() still be trusted? I think, at the least, more testing should be done with various configurations even after a suitable patch to DM is merged. What about PGSQL users using some kind of elastic hosting?

Given the craze in 'cloud' technology, its an important question to ask (and research).

Cheers,
--Tim

#30 lore raw
In reply to Greg Freemyer

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Greg Freemyer
Sent: Monday, March 16, 2009 5:36 PM
To: LVM general discussion and development
Subject: Re: [linux-lvm] fsync() and LVM

Quoted 32 lines · Les Mikesell
Quoted 5 lines · Greg Freemyer
Quoted 5 lines
Quoted 4 lines

Those are some very significant subsystems. �I have to believe filesystems have another way to implement fsync if barriers are not supported in the stack of block susbsystems.

Quoted 8 lines
Quoted 3 lines

If you can't get the completion status from the underlying layer, how can a
filesystem possibly implement it?

Barriers is a specific technology and they were just implemented in linux around 2005 I think. �(see documentation/barriers.txt)

Surely there was a mechanism in place before that.

I'm not sure that's a reasonable assumption.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Maybe this discussion needs to move to a filesystem list, since it is the filesystem that is responsible for making fsync() work even in the absence of barriers.

I though linux ended up doing a sync of the entire outstanding buffered data
for a partition with horrible performance, at least on ext3.

Quoted 6 lines

Yes, I understand fsync is horribly slow in ext3 and that may be the reason. �Supposedly much better in ext4. �Still if a userspace app calls fsync and in turn the filesystem does something really slow due to the lack of barriers, then this conversation should be about the poor performance of fsync() when using lvm (or mdraid, or drdb), not the total lack of fsync() support.

Quoted 7 lines

I haven't seen anyone claim yet that there is support for fsync(), which must return the status of the completion of the operation to the application. �If it does, then the discussion could turn to performance.

--
�Les Mikesell
� [email protected]

Is your specific interest to ext3? If so, I suggest you post a question there along the lines of:

Device Mapper does not support barriers if more than one physical device is in use by the LV. If I'm using ext3 on a LV and I call fsync() from user space, how is fsync() implemented. Or is it not?

The ext4 list is <[email protected]>. I see some ext3 stuff posted there, or it may have its own list.

Greg

Signature · 8 lines
========================================
So what happens if there is a database implemented directly on a Logical Volume, not File System involved at all?

Should the fsync man page describe what happens when used on each type of File System, Logical Volume, disk partition and /or combination?

-----
Thanks:
Jack Allen
#31 lore raw
In reply to Stuart D. Gathman
Quoted 3 lines · Stuart D. Gathman

That is clearly wrong - since fsync() isn't LVM's responsibility. I think they mean that fsync() can't garrantee that any writes are actually on the platter.

Even if the disk cache is in write-thru mode, that is.

Quoted 5 lines
Quoted 3 lines

that data doesn't even get to the controller, and it doesn't matter if the disks have write caches enabled or not. Or if they have battery backed caches. Please read the thread I linked. If what they say it's true,

That is clearly wrong. If writes don't work, nothing works.

It's the flush (= write NOW) supposedly not working, not the write. Writes happen, just later and potentially not in order. You seems to assume that fsync() is the only way to have the data written. That's not clearly the case, most userland processes just issue write(), never fsync(), and data gets written anyway, sooner or later.

Quoted 4 lines
Quoted 2 lines

you can't use LVM for anything that needs fsync(), including mail queues (sendmail), mail storage (imapd), as such. So I'd really like to know.

fsync() is a file system call that writes dirty buffers,

sure, but it's not the only way to have dirty pages flushed. There's a kernel thread that flushes them every since and then, and there's also memory pressure. So a broken fsync() can go unnoticed, you become aware of it if and only if:

1) you run some application that needs it (most don't even use it);
2) the system crashes (power loss);
3) you are unlucky enough to hit the window of vulnerability.

If any of these conditions is not met, you won't be aware of a mulfunctioning fsync().

But I think I understand what you mean: if the API to flush to physical storage is the same (used by fsync(), by pdflush, by the VM system) then you're right, everything is broken. But I've been using LVM for years now, I'm assuming that's not the case. :)

Quoted 3 lines

and then waits
for the physical writes to complete. It is only the waiting part that is broken.

Half-broken is broken. And the bigger issue here it's not even the delay. The issue is ordering. For a database, loosing the last transactions is bad enough, loosing transactions in the middle of the timeline is even worse.

For the mail subsystems, there's almost no ordering requirement, still loosing messages is no good.

---------------

Ehm, I've decided to write a small test program. My system is a Fedora 7, so nowhere recent. My setup:

/home is a LV, belonging to VG 'vg_data', whose only PV is /dev/md6. /dev/md6 is a RAID1 md device, whose members are /dev/sda10 and /dev/sdb10. /dev/sda and /dev/sdb are both Seagate ST3320620AS SATA disks.

The filesystem is EXT3, mounted with noatime,data=ordered.

The attached program writes the same block on a file N times (looping on lseek/write. Depending on how it's compiled, it issues a fdatasync() after each write.

Here are the results, for 32MB of data written:

$ time ./test_nosync
real    0m0.056s
user    0m0.004s
sys     0m0.052s

clearly, not disk activity here.

$ time ./test_sync
real    0m2.070s
user    0m0.002s
sys     0m0.152s

Now the same after hdparm -W0 /dev/sda; hdparm -W0 /dev/sdb:

$ time ./test_sync
real    1m16.431s
user    0m0.004s
sys     0m0.273s

These are 4096 "transactions" of size 8192, w/o the overhaed of allocating new blocks (it writes to the same block over and over). The first test is meaningless (they are never really committed). The second test, it's about 2000 transactions per second. Too many. In the third test, I got only about 50 transactions per second, which makes a lot of sense.

It seems to me that in my setup, disabling the caches on the disks does bring data to the platters, and that noone is "lying" about fsync.

Now I'm _really_ confused.

(the following isn't meaningful for the discussion)

For the curious of you (I was) I commented out the lseek(). For the _nosync version it's the same (1/2 a second).

For the _sync version, with -W1 I get:

$ time ./test_sync
real    0m48.816s
user    0m0.002s
sys     0m0.483s

and with -W0:

$ time ./test_sync
real    3m6.674s
user    0m0.006s
sys     0m0.526s

Since all the test were done deleting the file each time, I think what happens here is that the file is increasing in size, so fdatasync() each time triggers a write of the inode. It's two writes per loop. So I tried keeping the file around, having my test program write on preallocated blocks.

With -W1:

$ time ./test_sync
real    0m11.253s
user    0m0.001s
sys     0m0.244s

with -W0:

$ time ./test_sync
real    0m46.353s
user    0m0.005s
sys     0m0.249s

.TM.

/*
* compile with -DDO_FSYNC=1 and then with -DDO_FSYNC=0 */
#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#if !defined(DO_FSYNC)

# error "You must define DO_FSYNC"
#endif

#define MYBUFSIZ BUFSIZ
#define BYTES_TO_WRITE (32*1024*1024) /* 32MB */

int main(int argc, char *argv[])

{
	int fd, rc, i;
	char buf[MYBUFSIZ] = { '\0', };

	fd = open("testfile", O_WRONLY|O_CREAT, 0600);
	if (fd < 0) {
		perror("open");
		exit(1);
	}

	for (i = 0; i < (BYTES_TO_WRITE/MYBUFSIZ); i++) {
		rc = lseek(fd, 0, SEEK_SET);
		if (rc < 0) {
			perror("lseek");
			exit(1);
		}
		rc = write(fd, buf, sizeof(buf));
		if (rc < 0) {
			perror("write");
			exit(1);
		}
#if DO_FSYNC
		fdatasync(fd);
		if (rc < 0) {
			perror("fdatasync");
			exit(1);
		}
#endif
	} 
}
#32 lore raw
Quoted 4 lines · Marco Colombo

It seems to me that in my setup, disabling the caches on the disks does bring data to the platters, and that noone is "lying" about fsync.

Now I'm _really_ confused.

That's been my claim all along - that the broken fsync only affects on disk cache. LVM itself does not reorder writes in any way - it just fails to pass along the write barrier. fsync() does *start* writing the dirty buffers (implemented in the fs code). It just doesn't wait for the writes to finish getting to the platters. Apparently, it does wait for the write to get to the drive (but I'm not certain).

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#33 lore raw
Quoted 5 lines · Stuart D. Gathman
Quoted 3 lines · Marco Colombo

It seems to me that in my setup, disabling the caches on the disks does bring data to the platters, and that noone is "lying" about fsync.

Quoted 7 lines
Quoted 1 line

Now I'm _really_ confused.

That's been my claim all along - that the broken fsync only affects
on disk cache. LVM itself does not reorder writes in any way - it just
fails to pass along the write barrier. fsync() does *start* writing
the dirty buffers (implemented in the fs code). It just doesn't wait for the writes to finish getting to the platters. Apparently, it does wait for the write to get to the drive (but I'm not certain).

Given that fsync() is supposed to return the status of the completion of the physical write, that sounds broken to me. Do the LVM's in question here have more than one underlying device, and does it matter?

Signature · 2 lines
  Les Mikesell
   [email protected]
#34 lore raw
Quoted 9 lines · Les Mikesell
Quoted 7 lines · Stuart D. Gathman

That's been my claim all along - that the broken fsync only affects on disk cache. LVM itself does not reorder writes in any way - it just fails to pass along the write barrier. fsync() does *start* writing the dirty buffers (implemented in the fs code). It just doesn't wait for the writes to finish getting to the platters. Apparently, it does wait for the write to get to the drive (but I'm not certain).

Quoted 4 lines

Given that fsync() is supposed to return the status of the completion of the physical write, that sounds broken to me. Do the LVM's in question here have more than one underlying device, and does it matter?

According to my tests, you get a 50x speedup when you turn the cache on. It means that fsync is waiting for something to happen, and this "something" happens 50 times faster only when you turn the disk write-back cache on.

It seems to me that the only explanation is that fsync is waiting for disk I/O to complete (and not just to begin otherwise the time would be the same). With the cache enabled, the disk reports completion when the data is in the cache (write-back behaviour), with cache disabled it waits for the data to be on platters (write-thru behaviour).

.TM.

#35 lore raw
In reply to Stuart D. Gathman
Quoted 5 lines
Quoted 2 lines

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches.

But fsync is implemented using 'write barriers' - so fsync does not work?

After fsync, all data should be sent from the OS to the disk controller:

a.) this work perfectly using LVM?

b.) this does not work at all using LVM?

c.) it works when you use one single physical drive with LVM?

I am confused. The thread on the postfix list claims that it does not work at
all?

- Dietmar

#36 lore raw

[Please forgive double-posting, I'm not sure my previous attempt succeeded]

Quoted 12 lines · Dietmar Maurer
Quoted 4 lines
Quoted 2 lines

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches.

But fsync is implemented using 'write barriers' - so fsync does not work?

After fsync, all data should be sent from the OS to the disk controller:

a.) this work perfectly using LVM?

Quoted 7 lines

b.) this does not work at all using LVM?

c.) it works when you use one single physical drive with LVM?

I am confused. The thread on the postfix list claims that it does not work at
all?

Well, it's on the PostgreSQL list, not postfix. But it may affect postfix as well. Quoting postfix documentation:

Gory details: the Postfix mail queue requires that (1) the file system can rename a file to a near-by directory without changing the file's inode number, and that (2) mail is safely stored after fsync() of that file (not its parent directory) returns successfully, even when that file is renamed to a near-by directory at some later point in time.

If fsync() doen't work, point (2) is not fulfilled.

Please note: that on PostgreSQL list is not speculation. It comes from measurements. Benchmarks show too high transaction rates, just as if fsync() was disabled. The explanation (they provided) is that LVM does not honor fsync().

By some reading I've done I'm not sure. Is blkdev_issue_flush() we're talking about? Please see: http://lkml.org/lkml/2007/5/25/71
Is a LVM (well, device mapper) device still a "FLUSHABLE device" by that definition? Apparently it's ok not to support BIO_RW_BARRIER, as long as you support blkdev_issue_flush(). Has something changed since then?

How would you classify a LVM device? SAFE, FLUSHABLE, BARRIER or something else (UNSAFE)?

.TM.

#37 lore raw
In reply to Dietmar Maurer
Quoted 12 lines · Dietmar Maurer
Quoted 4 lines
Quoted 2 lines

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches.

But fsync is implemented using 'write barriers' - so fsync does not work?

After fsync, all data should be sent from the OS to the disk controller:

a.) this work perfectly using LVM?

Quoted 7 lines

b.) this does not work at all using LVM?

c.) it works when you use one single physical drive with LVM?

I am confused. The thread on the postfix list claims that it does not work at
all?

Everything will seem to work until you have an inconvenient crash or disk error. That is, data will be written normally - whether you fsync or not. The point of fsync() though, is for an application to confirm that the file is committed to stable media and will be recoverable even if the application (or OS) crashes or the system loses power. The correct next action of the application will depend on the return status of the fsync() operation (e.g., acknowledging receipt of a mail message, considering a database change to be committed, etc.). What I believe is happening is that fsync() always returns as though it were successful even though the underlying operations haven't completed. That's ummm..., optimistic at best. But, everything will still work (and more quickly) as long as the physical write of the file and associated directory metadata eventually succeeds. Realistically, for most things it doesn't matter because for critical data you still have to deal with the possibility of a disk write that succeeds being unreadable later for a variety of reasons - and the rest isn't critical anyway. However, it would be good to know exactly what to expect here.

Signature · 2 lines
 Les Mikesell
   [email protected]
#38 lore raw
Quoted 17 lines
Quoted 9 lines · Dietmar Maurer
Quoted 4 lines
Quoted 2 lines

Does that mean I should never use more than one device if I have applications depending on fsync (databases)?

It just means that write barriers won't get passed to the device. This is only a problem if the devices have write caches.

But fsync is implemented using 'write barriers' - so fsync does not work?

After fsync, all data should be sent from the OS to the disk

controller:

Quoted 6 lines

a.) this work perfectly using LVM?

b.) this does not work at all using LVM?

c.) it works when you use one single physical drive with LVM?

Please, can someone answer that questions?

- Dietmar