True fsync() in Linux (on IDE)
Hello,
I'm wondering is there any way in Linux to do proper fsync(), which makes sure data is written to the disk.
Currently on IDE devices one can see, fsync() only flushes data to the drive cache which is not enough for ACID guaranties database server must give.
There is solution just to disable drive write cache, but it seems to slowdown performance way to much.
I would be also happy enough with some global kernel option which would enable drive cache flush on fsync :)
Mac OS X also has this "optimization", but at least it provides an alternative flush method for Database Servers:
fcntl(fd, F_FULLFSYNC, NULL)
can be used instead of fsync() to get true fsync() behavior.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 11 lines · Peter Zaitsev
Hello,
I'm wondering is there any way in Linux to do proper
fsync(), which makes sure data is written to the disk.Currently on IDE devices one can see,
fsync()only flushes data to the drive cache which is not enough for ACID guaranties database server must give.There is solution just to disable drive write cache, but it seems to slowdown performance way to much.
Chris and I have working real fsync() with the barrier patches. I'll clean it up and post a patch for vanilla 2.6.5-rc today.
Signature · 1 line
Jens Axboe
Quoted 2 lines · Jens Axboe
Chris and I have working real
fsync()with the barrier patches. I'll clean it up and post a patch for vanilla 2.6.5-rc today.
This is good news.
The barrier stuff is long overdue^UI'm looking forward to this.
I'm using the term "TCQ" liberally although it may be inexact for older (parallel) ATA generations:
All these ATA fsync() vs. write cache issues have been open for much too long - no reproaches, but it's a pity we haven't been able to have data consistency for data bases and fast bulk writes (that need the write cache without TCQ) in the same drive for so long. I have seen Linux introduce TCQ for PATA early in 2.5, then drop it again. Similarly, FreeBSD ventured into TCQ for ATA but appears to have dropped it again as well.
May I ask that the information whether a particular driver (file system, hardware) supports write barriers be exposed in a standard way, for instance in the Kconfig help lines?
If I recall correctly from earlier patches, the barrier stuff is 1. command model (ATA vs. SCSI) specific and 2. driver and hardware specific and 3. requires that the file system knows how to use this properly.
Given that file systems have certain write ordering requirements if they are to be recoverable after a crash, I suspect Linux has _not_ been able to guarantee on-disk consistency for any time for years, which means that a crash in the wrong moment can kill the file system itself if the drive has reordered writes - only ext3 without write cache seems to behave better in this respect (data=ordered).
I would like to have a document that shows which file system, which chipset driver for PATA, which chipset driver for ATA, which low-level SCSI host adaptor driver, which file system support write barrier. We will probably also need to check if intermediate layers such as md and dm-mod propagate such information.
Given the necessary information, I can hack together a HTML document to provide this information; this offer has however not seen any response in the past. I am however not acquainted with the drivers and need information from the kernel hackers. Without such support, such a documentation effort is doomed.
BTW, I should very much like to be able to trace the low-level write information that goes out to the device, possibly including the payload
- something like tcpdump for the ATA or SCSI commands that are sent to the driver. Is such a facility available?
Signature · 3 lines
Matthias Andree Encrypt your mail: my GnuPG key ID is 0x052E7D95
(no subject)
unsubscribe
(btw - maybe you don't like to be cc'ed on kernel posts, but I do. it's lkml etiquette to do so, and it makes sure that I see your mail. otherwise I might not, especially true for bigger threads. so please, cc people. thanks)
Quoted 12 lines · Matthias Andree
Quoted 2 lines · Jens Axboe
Chris and I have working real
fsync()with the barrier patches. I'll clean it up and post a patch for vanilla 2.6.5-rc today.This is good news.
The barrier stuff is long overdue^UI'm looking forward to this.
I'm using the term "TCQ" liberally although it may be inexact for older (parallel) ATA generations:
Quoted 7 lines
All these ATA
fsync()vs. write cache issues have been open for much too long - no reproaches, but it's a pity we haven't been able to have data consistency for data bases and fast bulk writes (that need the write cache without TCQ) in the same drive for so long. I have seen Linux introduce TCQ for PATA early in 2.5, then drop it again. Similarly, FreeBSD ventured into TCQ for ATA but appears to have dropped it again as well.
That's because PATA TCQ sucks :-)
Quoted 3 lines
May I ask that the information whether a particular driver (file system, hardware) supports write barriers be exposed in a standard way, for instance in the
Kconfighelp lines?
Since reiser is the first implementation of it, it gets to chose how this works. Currently that's done by giving -o barrier=flush (=ordered used to exist as well, it will probably return - right now we just played with IDE).
Quoted 4 lines
If I recall correctly from earlier patches, the barrier stuff is 1. command model (ATA vs. SCSI) specific and 2. driver and hardware specific and 3. requires that the file system knows how to use this properly.
Yes.
Quoted 7 lines
Given that file systems have certain write ordering requirements if they are to be recoverable after a crash, I suspect Linux has _not_ been able to guarantee on-disk consistency for any time for years, which means that a crash in the wrong moment can kill the file system itself if the drive has reordered writes - only ext3 without write cache seems to behave better in this respect (data=ordered).
Quoted 5 lines
I would like to have a document that shows which file system, which chipset driver for PATA, which chipset driver for ATA, which low-level SCSI host adaptor driver, which file system support write barrier. We will probably also need to check if intermediate layers such as md and dm-mod propagate such information.
Only PATA core needs to support it, not the chipset drivers. md and dm aren't a difficult to implement now that unplug/congestion already iterates the device list and I added a blkdev_issue_flush() command.
Quoted 5 lines
Given the necessary information, I can hack together a HTML document to provide this information; this offer has however not seen any response in the past. I am however not acquainted with the drivers and need information from the kernel hackers. Without such support, such a documentation effort is doomed.
Usual approach - just start writing, it's a lot easier to get corrections (people seem to be several times more willing to point out your errors than give you recomendations for something you haven't started yet).
Quoted 4 lines
BTW, I should very much like to be able to trace the low-level write information that goes out to the device, possibly including the payload
- something like tcpdump for the ATA or SCSI commands that are sent to the driver. Is such a facility available?
No.
Signature · 1 line
Jens Axboe
Jens Axboe schrieb am 2004-03-18:
Quoted 9 lines
Quoted 7 lines
All these ATA
fsync()vs. write cache issues have been open for much too long - no reproaches, but it's a pity we haven't been able to have data consistency for data bases and fast bulk writes (that need the write cache without TCQ) in the same drive for so long. I have seen Linux introduce TCQ for PATA early in 2.5, then drop it again. Similarly, FreeBSD ventured into TCQ for ATA but appears to have dropped it again as well.That's because PATA TCQ sucks :-)
True. Few drives support it, and many of these you would not want to run in production...
Quoted 8 lines
Quoted 3 lines
May I ask that the information whether a particular driver (file system, hardware) supports write barriers be exposed in a standard way, for instance in the
Kconfighelp lines?Since reiser is the first implementation of it, it gets to chose how this works. Currently that's done by giving -o barrier=flush (=ordered used to exist as well, it will probably return - right now we just played with IDE).
This looks as though this was not the default and required the user to know what he's doing. Would it be possible to choose a sane default (like flush for ATA or ordered for SCSI when the underlying driver supports ordered tags) and leave the user just the chance to override this?
Quoted 1 line
Only PATA core needs to support it, not the chipset drivers. md and dm
Hum, I know the older Promise chips were blacklisted for PATA TCQ in FreeBSD. Might "ordered" cause situations where similar things happen to Linux? How about SCSI/libata? Is the situation the same there?
Quoted 2 lines
aren't a difficult to implement now that unplug/congestion already iterates the device list and I added a
blkdev_issue_flush()command.
So this would - for SCSI - be an sd issue rather than a driver issue as well?
Signature · 3 lines
Matthias Andree Encrypt your mail: my GnuPG key ID is 0x052E7D95
Quoted 8 lines · Matthias Andree
Quoted 8 lines
Quoted 7 lines
All these ATA
fsync()vs. write cache issues have been open for much too long - no reproaches, but it's a pity we haven't been able to have data consistency for data bases and fast bulk writes (that need the write cache without TCQ) in the same drive for so long. I have seen Linux introduce TCQ for PATA early in 2.5, then drop it again. Similarly, FreeBSD ventured into TCQ for ATA but appears to have dropped it again as well.
Quoted 4 lines
Quoted 1 line
That's because PATA TCQ sucks :-)
True. Few drives support it, and many of these you would not want to run in production...
Plus, the spec is broken.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
May I ask that the information whether a particular driver (file system, hardware) supports write barriers be exposed in a standard way, for instance in the
Kconfighelp lines?Since reiser is the first implementation of it, it gets to chose how this works. Currently that's done by giving -o barrier=flush (=ordered used to exist as well, it will probably return - right now we just played with IDE).
Quoted 5 lines
This looks as though this was not the default and required the user to know what he's doing. Would it be possible to choose a sane default (like flush for ATA or ordered for SCSI when the underlying driver supports ordered tags) and leave the user just the chance to override this?
When things have matured, might not be a bad idea to default to using barriers.
Quoted 5 lines
Quoted 1 line
Only PATA core needs to support it, not the chipset drivers. md and dm
Hum, I know the older Promise chips were blacklisted for PATA TCQ in FreeBSD. Might "ordered" cause situations where similar things happen to Linux? How about SCSI/libata? Is the situation the same there?
Don't confuse TCQ and barriers, it has nothing to do with each other for IDE. I can't imagine any chipsets having problems with a syncronize cache command.
Quoted 5 lines
Quoted 2 lines
aren't a difficult to implement now that unplug/congestion already iterates the device list and I added a
blkdev_issue_flush()command.So this would - for SCSI - be an sd issue rather than a driver issue as well?
No, for scsi it's a low level driver issue. IDE chipset 'drivers' really aren't anything but setup stuff, and maybe a few hooks to deal with dma. All the action is in the ide core.
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
Quoted 2 lines
There is solution just to disable drive write cache, but it seems to slowdown performance way to much.
Chris and I have working real
fsync()with the barrier patches. I'll clean it up and post a patch for vanilla 2.6.5-rc today.
Good to hear. How is it going to work from user point of view ? Just fsync working back again or there would be some special handling.
Also. What is about fsync() in 2.6 nowadays ?
I've done some tests on 3WARE RAID array and it looks like it is different compared to 2.4 I've been testing previously.
I have the simple test which has single page writes to the file followed
by fsync(). First run give you the case when file grows with each
write, second when you're writing to existing file space.
The results I have on 2.4 is something like 40 sec per 1000 fsyncs for new file, and 0.6 sec for existing file.
With 2.6.3 I have both existing file and new file to complete in less than 1 second.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 10 lines · Peter Zaitsev
Quoted 5 lines · Jens Axboe
Quoted 2 lines
There is solution just to disable drive write cache, but it seems to slowdown performance way to much.
Chris and I have working real
fsync()with the barrier patches. I'll clean it up and post a patch for vanilla 2.6.5-rc today.Good to hear. How is it going to work from user point of view ? Just fsync working back again or there would be some special handling.
It's just going to work :)
Quoted 9 lines
Also. What is about
fsync()in 2.6 nowadays ?I've done some tests on 3WARE RAID array and it looks like it is different compared to 2.4 I've been testing previously.
I have the simple test which has single page writes to the file followed
byfsync(). First run give you the case when file grows with each
write, second when you're writing to existing file space.
Quoted 5 lines
The results I have on 2.4 is something like 40 sec per 1000 fsyncs for new file, and 0.6 sec for existing file.
With 2.6.3 I have both existing file and new file to complete in less than 1 second.
I believe some missed set_page_writeback() calls caused fsync() to never really wait on anything, pretty broken... IIRC, it's fixed in latest -mm, or maybe it's just pending for next release.
Signature · 1 line
Jens Axboe
Quoted 6 lines · Jens Axboe
Quoted 2 lines
With 2.6.3 I have both existing file and new file to complete in less than 1 second.
I believe some missed
set_page_writeback()calls causedfsync()to never really wait on anything, pretty broken... IIRC, it's fixed in latest -mm, or maybe it's just pending for next release.
This should have only been broken in -mm. Which kernels exactly are you comparing? Maybe the 3ware array defaults to different writecache settings under 2.6?
-chris
Quoted 7 lines · Chris Mason
Quoted 3 lines
I believe some missed
set_page_writeback()calls causedfsync()to never really wait on anything, pretty broken... IIRC, it's fixed in latest -mm, or maybe it's just pending for next release.This should have only been broken in -mm. Which kernels exactly are you comparing? Maybe the 3ware array defaults to different writecache settings under 2.6?
I'm trying RH AS 3.0 kernel, however I have the same behavior on my SuSE 8.2 workstation.
I use 2.6.3 kernel for tests now (It is not the latest I know) EXT3 file system.
3WARE has writeback cache setting in both cases.
Here is the test program I was using:
#include <stdio.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
char buffer[4096] __attribute__((__aligned__(4096)));
main()
{
int rc2,rc;
int i;
buffer[0]=(char)getpid(); rc=open("write",O_RDWR | O_CREAT,0666); if (rc==-1) printf("Error at open: %d\n",errno); for(i=0;i<1000;i++)
{
rc2=write(rc,&buffer,4096);
printf(".");
fflush(stdout);
if (rc2<0)
{
printf("Error code: %d\n",errno);
return;
} fsync(rc);
}
}
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 6 lines · Peter Zaitsev
Quoted 4 lines · Chris Mason
Quoted 3 lines
I believe some missed
set_page_writeback()calls causedfsync()to never really wait on anything, pretty broken... IIRC, it's fixed in latest -mm, or maybe it's just pending for next release.
Quoted 7 lines
Quoted 3 lines
This should have only been broken in -mm. Which kernels exactly are you comparing? Maybe the 3ware array defaults to different writecache settings under 2.6?
I'm trying RH AS 3.0 kernel, however I have the same behavior on my SuSE 8.2 workstation.
Some suse 8.2 kernels had write barriers for IDE, some did not. If you're running any kind of recent suse kernel, you're doing cache flushes on fsync with ext3.
Not sure if RH has ever carried the patches or not. Easy enough to test for on suse, just look for blk_queue_ordered in the System.map.
Quoted 4 lines
I use 2.6.3 kernel for tests now (It is not the latest I know) EXT3 file system.
3WARE has writeback cache setting in both cases.
Then it sounds like your 2.4 is doing flushes. I'd expect this test to run very quickly without them.
-chris
Quoted 3 lines · Chris Mason
Some suse 8.2 kernels had write barriers for IDE, some did not. If you're running any kind of recent suse kernel, you're doing cache flushes on fsync with ext3.
I have this kernel:
Linux abyss 2.4.20-4GB #1 Sat Feb 7 02:07:16 UTC 2004 i686 unknown unknown GNU/Linux
I believe it is reasonably recent one from Hubert's kernels.
The thing is the performance is different if file grows or it does not. If it does - we have some 25 fsync/sec. IF we're writing to existing one, we have some 1600 fsync/sec
In the former case cache is surely not flushed.
Quoted 7 lines
Quoted 4 lines
I use 2.6.3 kernel for tests now (It is not the latest I know) EXT3 file system.
3WARE has writeback cache setting in both cases.
Then it sounds like your 2.4 is doing flushes. I'd expect this test to run very quickly without them.
2.4 does flush in one case but not in other. 2.6 does not do it in ether case.
I was also surprised to see this simple test case has so different
performance with default and "deadline" IO scheduler - 1.6 vs 0.5 sec
per 1000 fsync's.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 12 lines · Peter Zaitsev
Quoted 3 lines · Chris Mason
Some suse 8.2 kernels had write barriers for IDE, some did not. If you're running any kind of recent suse kernel, you're doing cache flushes on fsync with ext3.
I have this kernel:
Linux abyss 2.4.20-4GB #1 Sat Feb 7 02:07:16 UTC 2004 i686 unknown unknown GNU/Linux
Quoted 8 lines
I believe it is reasonably recent one from Hubert's kernels.
The thing is the performance is different if file grows or it does not. If it does - we have some 25 fsync/sec. IF we're writing to existing one, we have some 1600 fsync/sec
In the former case cache is surely not flushed.
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Thanks to Jens, the 2.6 barrier patch has a nice clean way to allow barriers on fsync, O_SYNC, O_DIRECT, etc, so we can make IDE drives much safer than the 2.4 code did.
I had a patch to make fsync always generate the barriers in 2.4, but it was tricky since it had to figure out the last buffer it was going to write before it wrote it. The 2.6 code is much better.
Quoted 6 lines
2.4 does flush in one case but not in other. 2.6 does not do it in ether case.
I was also surprised to see this simple test case has so different
performance with default and "deadline" IO scheduler - 1.6 vs 0.5 sec
per 1000 fsync's.
Not sure on that one, both cases are generating tons of unplugs, the drive is just responding insanely fast.
-chris
Quoted 5 lines · Chris Mason
Quoted 2 lines
In the former case cache is surely not flushed.
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Oh. Yes. This is Reiser, I did not think it is FS issue. I'll know to stay away from ReiserFS now.
Quoted 4 lines
Thanks to Jens, the 2.6 barrier patch has a nice clean way to allow barriers on fsync,
O_SYNC,O_DIRECT, etc, so we can make IDE drives much safer than the 2.4 code did.
Great.
Quoted 6 lines
Quoted 3 lines
I was also surprised to see this simple test case has so different
performance with default and "deadline" IO scheduler - 1.6 vs 0.5 sec
per 1000 fsync's.Not sure on that one, both cases are generating tons of unplugs, the drive is just responding insanely fast.
Well why it would be slow if it has write cache off.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 10 lines · Peter Zaitsev
Quoted 5 lines · Chris Mason
Quoted 2 lines
In the former case cache is surely not flushed.
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Oh. Yes. This is Reiser, I did not think it is FS issue. I'll know to stay away from ReiserFS now.
For reiserfs data=ordered should be enough to trigger the needed commits. If not, data=journal. Note that neither fs does barriers for O_SYNC, so we're just not perfect in 2.4.
-chris
Quoted 29 lines · Chris Mason
Quoted 8 lines · Peter Zaitsev
Quoted 4 lines · Chris Mason
Quoted 4 lines
In the former case cache is surely not flushed.
Quoted 9 lines
Quoted 5 lines
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Oh. Yes. This is Reiser, I did not think it is FS issue. I'll know to stay away from ReiserFS now.
For reiserfs data=ordered should be enough to trigger the needed commits. If not, data=journal. Note that neither fs does barriers for
O_SYNC, so we're just not perfect in 2.4.(quoted diff) +0
-2-chris -
Quoted 8 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
You are not listening to Peter. As I understand it from what Peter says and your words, your implementation is wrong, and makes fsync meaningless. If so, then you need to fix it. fsync should not be meaningless even for metadata only journaling. This is a serious bug that needs immediate correction, if Peter and I understand it correctly from your words.
Signature · 1 line
Hans
Quoted 27 lines · Hans Reiser
Quoted 25 lines · Chris Mason
Quoted 8 lines · Peter Zaitsev
Quoted 4 lines · Chris Mason
Quoted 4 lines
In the former case cache is surely not flushed.
Quoted 9 lines
Quoted 5 lines
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Oh. Yes. This is Reiser, I did not think it is FS issue. I'll know to stay away from ReiserFS now.
For reiserfs data=ordered should be enough to trigger the needed commits. If not, data=journal. Note that neither fs does barriers for
O_SYNC, so we're just not perfect in 2.4.
Quoted 8 lines
Quoted 2 lines
-chris
You are not listening to Peter. As I understand it from what Peter says and your words, your implementation is wrong, and makes fsync meaningless. If so, then you need to fix it. fsync should not be meaningless even for metadata only journaling. This is a serious bug that needs immediate correction, if Peter and I understand it correctly from your words.
I am listening to Peter, Jens and I have spent a significant amount of time on this code. We can go back and spend many more hours testing and debugging the 2.4 changes, or we can go forward with a very nice solution in 2.6.
I'm planning on going forward with 2.6
-chris
Quoted 6 lines · Chris Mason
I am listening to Peter, Jens and I have spent a significant amount of time on this code. We can go back and spend many more hours testing and debugging the 2.4 changes, or we can go forward with a very nice solution in 2.6.
I'm planning on going forward with 2.6
Chris, Hans
It is great to hear this is going to be fixed in 2.6, however it is quite a pity we have a real mess with this in 2.4 series.
Resuming what I've heard so far it looks like it depends on:
- If it is fsync/O_SYNC or O_DIRECT (which user would expect to have
the same effect in this respect.
- It depends on kernel version. Some vendors have some fixes, while others do not have them.
- It depends on hardware - if it has write cache on or off
- It depends on type of write (if it changes mata data or not)
- Finally it depends on file system and even journal mount options
Just curious does at least Asynchronous IO have the same behavior as standard IO ?
All of these makes it extremely hard to explain what do users need in order to get durability for their changes, while preserving performance.
Furthermore as it was broken for years I expect we'll have people which developed things with fast fsync() in mind, who would start screaming once we have real fsync()
(see my mail about Apple actually disabling cache flush on fsync() due to this reason)
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 8 lines · Peter Zaitsev
Quoted 5 lines · Chris Mason
I am listening to Peter, Jens and I have spent a significant amount of time on this code. We can go back and spend many more hours testing and debugging the 2.4 changes, or we can go forward with a very nice solution in 2.6.
Quoted 7 lines
Quoted 1 line
I'm planning on going forward with 2.6
Chris, Hans
It is great to hear this is going to be fixed in 2.6, however it is quite a pity we have a real mess with this in 2.4 series.
It is indeed.
Quoted 10 lines
Resuming what I've heard so far it looks like it depends on:
- If it is fsync/O_SYNC or
O_DIRECT(which user would expect to have
the same effect in this respect.
- It depends on kernel version. Some vendors have some fixes, while others do not have them.
- It depends on hardware - if it has write cache on or off
- It depends on type of write (if it changes mata data or not)
- Finally it depends on file system and even journal mount options
All of the above is correct.
Quoted 3 lines
Just curious does at least Asynchronous IO have the same behavior as standard IO ?
For the suse patch, yes. If it triggers a commit, you get a cache flush.
Quoted 10 lines
All of these makes it extremely hard to explain what do users need in order to get durability for their changes, while preserving performance.
Furthermore as it was broken for years I expect we'll have people which developed things with fast
fsync()in mind, who would start screaming once we have realfsync()(see my mail about Apple actually disabling cache flush on
fsync()due to this reason)
These are all difficult issues. I wish I had easier answers for you, hopefully we can get it all nailed down in 2.6 for starters.
-chris
Quoted 9 lines · Chris Mason
Quoted 6 lines
- It depends on type of write (if it changes mata data or not)
- Finally it depends on file system and even journal mount optionsAll of the above is correct.
Doesn't the above statement contradict the following?:
Quoted 4 lines
In 2.6 does fsync always insert a write barrier when the metadata
Quoted 1 line
journaling option is set for reiserfs?
Yes, fsync is done in the 2.6 patches.
and I was imprecise, I should have asked, does fsync flush the disk cache regardless of what mount options are set or data/metadata touched in the 2.6 patches?
Signature · 1 line
Hans
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Chris Mason
Quoted 6 lines
- It depends on type of write (if it changes mata data or not)
- Finally it depends on file system and even journal mount options
Quoted 5 lines
Quoted 3 lines
All of the above is correct.
Doesn't the above statement contradict the following?:
Sorry for the confusion, I thought he was asking about 2.4.x.
Quoted 11 lines
Quoted 4 lines
In 2.6 does fsync always insert a write barrier when the metadata
Quoted 1 line
journaling option is set for reiserfs?
Yes, fsync is done in the 2.6 patches.
and I was imprecise, I should have asked, does fsync flush the disk cache regardless of what mount options are set or data/metadata touched in the 2.6 patches?
-chris
Quoted 6 lines · Chris Mason
Quoted 6 lines
and I was imprecise, I should have asked, does fsync flush the disk cache regardless of what mount options are set or data/metadata touched in the 2.6 patches?
Forgive my relentlessness, is the answer to the above yes?
Signature · 1 line
Hans
Quoted 9 lines · Hans Reiser
Quoted 5 lines · Chris Mason
Quoted 5 lines
and I was imprecise, I should have asked, does fsync flush the disk cache regardless of what mount options are set or data/metadata touched in the 2.6 patches?
Forgive my relentlessness, is the answer to the above yes?
Yes ;-) One goal of the 2.6 patches is to make it possible for higher levels to easily insert flushes when needed. The reiserfs fsync and ext3 fsync code will both do this.
I realized I wasn't clear about device mapper and md earlier, both need extra work to aggregate the flushes down to the drives. They don't yet support flushing.
-chris
Quoted 13 lines · Chris Mason
Quoted 7 lines · Hans Reiser
Quoted 3 lines · Chris Mason
Quoted 3 lines
and I was imprecise, I should have asked, does fsync flush the disk cache regardless of what mount options are set or data/metadata touched in the 2.6 patches?
Quoted 9 lines
Quoted 3 lines
Forgive my relentlessness, is the answer to the above yes?
Yes ;-) One goal of the 2.6 patches is to make it possible for higher levels to easily insert flushes when needed. The reiserfs fsync and ext3 fsync code will both do this.
will do it, or do do it?
Quoted 11 lines
I realized I wasn't clear about device mapper and md earlier, both need extra work to aggregate the flushes down to the drives. They don't yet support flushing.
-chris
Sounds like things are very much unfinished and you guys need more time before these patches are ready for inclusion. We need to get reiser4 to use this stuff. Zam, put that in your todo list.
Signature · 1 line
Hans
Quoted 45 lines · Chris Mason
Quoted 42 lines · Hans Reiser
Quoted 38 lines · Chris Mason
Quoted 19 lines · Peter Zaitsev
Quoted 6 lines · Chris Mason
Quoted 6 lines
In the former case cache is surely not flushed.
Quoted 7 lines
Hmmm, is it reiser? For both 2.4 reiserfs and ext3, the flush happens when you commit. ext3 always commits on fsync and reiser only commits when you've changed metadata.
Quoted 6 lines
Oh. Yes. This is Reiser, I did not think it is FS issue. I'll know to stay away from ReiserFS now.
For reiserfs data=ordered should be enough to trigger the needed commits. If not, data=journal. Note that neither fs does barriers for
O_SYNC, so we're just not perfect in 2.4.-chris
Quoted 7 lines
Quoted 3 lines
You are not listening to Peter. As I understand it from what Peter says and your words, your implementation is wrong, and makes fsync meaningless. If so, then you need to fix it. fsync should not be meaningless even for metadata only journaling. This is a serious bug that needs immediate correction, if Peter and I understand it correctly from your words.
I am listening to Peter, Jens and I have spent a significant amount of time on this code.
but you need to get it right.
Quoted 7 lines
We can go back and spend many more hours testing and debugging the 2.4 changes, or we can go forward with a very nice solution in 2.6.
I'm planning on going forward with 2.6
This is a very important patch that you have created, but you haven't articulated what happens in the following scenario (Peter I am making up something without knowing your internals, please feel encouraged to help me on this).
mysql fsync()'s a file, which it thinks guarantees that all of a mysql transaction has reached disk. The disk write caches it. You let fsync return. It is not on disk. mysql performs its mysql commit, and writes a mysql commit record which reaches disk, but not all of the transaction is on disk. The system crashes. mysql plays the log. mysql has internal corruption. User calls Peter. Peter asks, what do you expect when you use a piece of shit like reiserfs? User doesn't care about our internal squabbling and goes back to using windows which does proper commits.
Or, random application fsyncs, expects that it means that data has reached disk, and tells user to perform real world actions dependent on the data being on disk, but it is not.
I hope I am totally off-base and not understanding you.... Please help me here.
Quoted 12 lines
(quoted diff) +0
-2-chris -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
Hans
Quoted 2 lines · Hans Reiser
I hope I am totally off-base and not understanding you.... Please help me here.
Lets look at actual scope of the problem:
filesystem metadata
filesystem data (fsync, O_SYNC, O_DIRECT)
block device data (fsync, O_SYNC, O_DIRECT)
Multiply the cases above times each filesystem and also times md and device mapper, since the barriers need to aggregate down to all the drives.
In other words, just fixing fsync in 2.4 is not enough, and there is still considerable development needed in 2.6. Maybe after all the 2.6 changes are done and accepted we can consider backporting parts of it to 2.4.
-chris
Quoted 18 lines · Chris Mason
Quoted 3 lines · Hans Reiser
I hope I am totally off-base and not understanding you.... Please help me here.
Lets look at actual scope of the problem:
filesystem metadata
filesystem data (fsync,O_SYNC,O_DIRECT)
block device data (fsync,O_SYNC,O_DIRECT)Multiply the cases above times each filesystem and also times md and device mapper, since the barriers need to aggregate down to all the drives.
Quoted 12 lines
In other words, just fixing fsync in 2.4 is not enough, and there is still considerable development needed in 2.6. Maybe after all the 2.6 changes are done and accepted we can consider backporting parts of it to 2.4.
-chris
In 2.6 does fsync always insert a write barrier when the metadata journaling option is set for reiserfs?
Signature · 1 line
Hans
Quoted 11 lines · Hans Reiser
Quoted 10 lines · Chris Mason
Lets look at actual scope of the problem:
filesystem metadata
filesystem data (fsync,O_SYNC,O_DIRECT)
block device data (fsync,O_SYNC,O_DIRECT)Multiply the cases above times each filesystem and also times md and device mapper, since the barriers need to aggregate down to all the drives.
Quoted 7 lines
Quoted 5 lines
In other words, just fixing fsync in 2.4 is not enough, and there is still considerable development needed in 2.6. Maybe after all the 2.6 changes are done and accepted we can consider backporting parts of it to 2.4.
In 2.6 does fsync always insert a write barrier when the metadata journaling option is set for reiserfs?
Yes, fsync is done in the 2.6 patches. O_SYNC, O_DIRECT and others are not yet. The important part right now is to get the IDE core bits reviewed and all the FS guys to agree on how we want to use them.
It's much cleaner in 2.6, the filesystem can just request a flush after the last data buffer goes down the pipe.
-chris
Quoted 9 lines · Hans Reiser
mysql fsync()'s a file, which it thinks guarantees that all of a mysql transaction has reached disk. The disk write caches it. You let fsync return. It is not on disk. mysql performs its mysql commit, and writes a mysql commit record which reaches disk, but not all of the transaction is on disk. The system crashes. mysql plays the log. mysql has internal corruption. User calls Peter. Peter asks, what do you expect when you use a piece of shit like reiserfs? User doesn't care about our internal squabbling and goes back to using windows which does proper commits.
This is right,
We had some unexplained data corruptions in Innodb which can be explained by broken fsync(), but in the most cases the scenario is less gloomy. Users just do not see some of last committed transactions if they test durability by shutting off the power, which is however already not good enough for critical applications.
However this is due to external pre-caution Innodb does. It uses "double write buffer", which basically means each page is first written to some small page based log file, and only afterwards written to the
proper place on the disk. We have to do it even with proper fsync()
implementation as there is still possibility to crash in the middle of fsync (or synchronous write) which will result in partial page write. Think for example about the case when page crosses stripe boundary on RAID.
If file system would guaranty atomicity of write() calls (synchronous would be enough) we could disable it and get good extra performance.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/
Quoted 2 lines · Peter Zaitsev
If file system would guaranty atomicity of
write()calls (synchronous would be enough) we could disable it and get good extra performance.
Berkeley DB 4.2.52 for instance documents that page writes (of data base pages) must be atomic, hence, if the data base page size is larger than what the FS can write atomically, a crash may leave the data base in a non-recoverable (catastrophic) state. (This assumes using the write-ahead logging "Berkeley DB Transactional Data Store" mode of operation, the other modes aren't recoveable after crash anyways.)
Signature · 3 lines
Matthias Andree Encrypt your mail: my GnuPG key ID is 0x052E7D95
Quoted 2 lines · Peter Zaitsev
If file system would guaranty atomicity of
write()calls (synchronous would be enough) we could disable it and get good extra performance.
Store an MD5 or SHA digest of the page in the page itself, or elsewhere. (Obviously the digest doesn't include the bytes used to store it).
Then partial write errors are always detectable, even if there's a hardware failure, so journal writes are effectively atomic.
-- Jamie
Quoted 9 lines · Jamie Lokier
Quoted 2 lines · Peter Zaitsev
If file system would guaranty atomicity of
write()calls (synchronous would be enough) we could disable it and get good extra performance.Store an MD5 or SHA digest of the page in the page itself, or elsewhere. (Obviously the digest doesn't include the bytes used to store it).
Then partial write errors are always detectable, even if there's a hardware failure, so journal writes are effectively atomic.
Jamie,
The problem is not detecting the partial page writes, but dealing with
them. Obviously there is checksum on the page (it is however not
MD5/SHA which are designed for cryptographic needs) and so page corruption is detected if it happens for whatever reason.
The problem is you can't do anything with the page if only unknown portion of it was modified.
Innodb uses sort of "logical" logging which just says something like
delete row #2 from page #123, so if page is badly corrupted it will not help to recover.
Of course you can log full pages, but this will increase overhead significantly, especially for small row sizes.
This is why solution now is to use long term "logical" log and short term "physical" log, which is used by background page writer, before writing pages to their original locations.
Signature · 5 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com Meet the MySQL Team at User Conference 2004! (April 14-16, Orlando,FL) http://www.mysql.com/uc2004/