Skip to contentSkip to conversation

True fsync() in Linux (on IDE)

DISCUSSION Heikki Tuuri 6 participants 7 messages Mar 22, 2004

lore
Share & tools
lore
#0 lore raw

Hi!

I have written the InnoDB backend to MySQL. Some notes on the fsync() processing problem:

1. It is dangerous for a database if fsync'ed files are physically written to the disk in an order different from the order in which the fsync's were called on them. In a power outage this can cause database corruption.

For example, a database must make sure that the log file is written to the disk at least up to the 'log sequence number' of any data page written to disk. Thus, we must first write to the log file and call fsync() on it, and only after that are allowed to write the data page to a data file and call fsync() on the data file.

2. An 'atomic' file write in the OS does not solve the problem of partially written database pages in a power outage if the disk drive is not guaranteed to stay operational long enough to be able to write the whole page physically to disk. An InnoDB data page is 16 kB, and probably not guaranteed to be any 'atomic' unit of physical disk writes. However, in practice, half-written pages (either because of the OS or the disk) seem to be very rare.

3. Jeffrey Siegal wrote to me that he checked a few disk drives if they support a cache flush. Some of them did, others did not. If the disk drive does not support a cache flush, then the only way to do a proper fsync is to configure it not to cache writes at all. Though, in some drives even the non-cache configuration option may be missing.

Best regards,

Heikki Tuuri
Innobase Oy
http://www.innodb.com

...........

List:       linux-kernel
Subject:    Re: True  fsync() in Linux (on IDE)
From:       Peter Zaitsev <peter () mysql ! com>
Date:       2004-03-20 19:48:23
Message-ID: <1079812102.3182.31.camel () abyss ! local>

[Download message RAW]

Quoted 9 lines · Jamie Lokier
Quoted 2 lines · Peter Zaitsev

If file system would guaranty atomicity of write() calls (synchronous would be enough) we could disable it and get good extra performance.

Store an MD5 or SHA digest of the page in the page itself, or elsewhere. (Obviously the digest doesn't include the bytes used to store it).

Then partial write errors are always detectable, even if there's a hardware failure, so journal writes are effectively atomic.

Jamie,

The problem is not detecting the partial page writes, but dealing with
them. Obviously there is checksum on the page (it is however not
MD5/SHA which are designed for cryptographic needs) and so page corruption is detected if it happens for whatever reason.

The problem is you can't do anything with the page if only unknown portion of it was modified.

Innodb uses sort of "logical" logging which just says something like
delete row #2 from page #123, so if page is badly corrupted it will not help to recover.

Of course you can log full pages, but this will increase overhead significantly, especially for small row sizes.

This is why solution now is to use long term "logical" log and short term "physical" log, which is used by background page writer, before writing pages to their original locations.

Signature · 2 lines
Peter Zaitsev, Senior Support Engineer
MySQL AB, www.mysql.com
#1 lore raw
Quoted 7 lines · Heikki Tuuri

2. An 'atomic' file write in the OS does not solve the problem of partially written database pages in a power outage if the disk drive is not guaranteed to stay operational long enough to be able to write the whole page physically to disk. An InnoDB data page is 16 kB, and probably not guaranteed to be any 'atomic' unit of physical disk writes. However, in practice, half-written pages (either because of the OS or the disk) seem to be very rare.

There's no such thing as atomic writes bigger than a sector really, we just pretend there is. Timing usually makes this true.

For bigger atomic writes, 2.4 SUSE kernel had some nasty hack (called blk-atomic) to prevent reordering by the io scheduler to avoid partial blocks from databases. For 2.6 it's much easier since you can just send a 16kb write out as one unit. So you send out the db data page as 1 hardware request, which is pretty much as atomic as you can do it from the OS point of view. As long as you avoid a big window between parts of this data page, you've narrowed the window of breakage from many seconds to miliseconds (maybe even 0, if drive platter inertia guarentees you that the write will ensure the data is on platter).

Signature · 1 line
Jens Axboe
#2 lore raw

Jens Axboe schrieb am 2004-03-22:

Quoted 2 lines

There's no such thing as atomic writes bigger than a sector really, we just pretend there is. Timing usually makes this true.

If there is no such atomicity (except maybe in ext3fs data=journal or the upcoming reiserfs4 - isn't there?), then nobody should claim so. If the kernel cannot 100.00000000% guarantee the write is atomic, claiming otherwise is plain fraud and nothing else.

Some people bet their whole business/company and hence a fair deal of their belongings on a single data base, and making them believe facts that simply aren't reality is dangerous. These people will have very little understanding for sloppiness here. Linux has no obligation to be fast or reliable, but it MUST PROPERLY AND TRUTHFULLY state what it can guarantee and what it cannot guarantee.

Quoted 3 lines

For bigger atomic writes, 2.4 SUSE kernel had some nasty hack (called blk-atomic) to prevent reordering by the io scheduler to avoid partial blocks from databases.

That does not make a write atomic if the scheduled blocks are still written one at a time (and I believe tagged command queueing won't help to unroll partial writes either).

If the hardware support is missing, it is prudent to say just that and not make any bogus promises about platter inertia and "it usually works". (who says that the filter curves adjust to the decreasing platter speed and the electronics are sustained for long enough? how about write verify and remapping broken blocks?)

So we only write one hardware block size atomically, usually 512 bytes on ATA and SCSI disk drives (MO might do 2048 at a time, but why introduce complexity). That's a data point in this whole fsync() discussion.

Signature · 3 lines
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95
#3 lore raw
Quoted 4 lines · Matthias Andree

If there is no such atomicity (except maybe in ext3fs data=journal or the upcoming reiserfs4 - isn't there?), then nobody should claim so. If the kernel cannot 100.00000000% guarantee the write is atomic, claiming otherwise is plain fraud and nothing else.

Who claims writes are atomic?

#4 lore raw
In reply to Matthias Andree
Quoted 5 lines · Matthias Andree

Jens Axboe schrieb am 2004-03-22:

Quoted 2 lines

There's no such thing as atomic writes bigger than a sector really, we just pretend there is. Timing usually makes this true.

;)

Quoted 6 lines

If there is no such atomicity (except maybe in ext3fs data=journal or the upcoming reiserfs4 - isn't there?), then nobody should claim so. If the kernel cannot 100.00000000% guarantee the write is atomic, claiming otherwise is plain fraud and nothing else.

Quoted 6 lines

Some people bet their whole business/company and hence a fair deal of their belongings on a single data base, and making them believe facts that simply aren't reality is dangerous. These people will have very little understanding for sloppiness here. Linux has no obligation to be fast or reliable, but it MUST PROPERLY AND TRUTHFULLY state what it can guarantee and what it cannot guarantee.

Some databases (eg. oracle) can write a checksum for each database page to overcome this problem, as this is not just "a linux problem".

Signature · 4 lines
	Christoffer

	Topper Harley: Interesting perfume.
	Ramada Thompson: It's Vicks. I have a cold.
#5 lore raw
Quoted 12 lines · Christoffer Hall-Frederiksen
Quoted 11 lines

If there is no such atomicity (except maybe in ext3fs data=journal or the upcoming reiserfs4 - isn't there?), then nobody should claim so. If the kernel cannot 100.00000000% guarantee the write is atomic, claiming otherwise is plain fraud and nothing else.

Some people bet their whole business/company and hence a fair deal of their belongings on a single data base, and making them believe facts that simply aren't reality is dangerous. These people will have very little understanding for sloppiness here. Linux has no obligation to be fast or reliable, but it MUST PROPERLY AND TRUTHFULLY state what it can guarantee and what it cannot guarantee.

Quoted 2 lines

Some databases (eg. oracle) can write a checksum for each database page to overcome this problem, as this is not just "a linux problem".

I am aware some databases support checksumming (Berkeley DB also does, since v4.1 (*), and probably a lot more so they know where log recovery starts) but does that make statements sensible that claim the timing (some stochastic factor) would usually give "guarantees" about atomicity of the individual page write when the hardware doesn't guarantee anything beyond 512 bytes at a time? I think it does not.

I don't mind to beat up anyone, I'd just like to have the guarantees documented without thin-ice kind of promises "usually you'll get more".

It's good to get more than what was asked for, but the application designer cannot take that into account because he gets no guarantees. So why bother wasting space and time for writing and reading such lines? Or even discussing?

Maybe some interface so an application can query the maximum size of an atomic write for any given file system (stat[v]fs extension maybe) would be useful though, so applications can be optimized for data-journaling file systems should these prove capable to provide "large atomic write" guarantees.

(*) http://cvs.sourceforge.net/viewcvs.py/bogofilter/bogofilter/src/datastore_db.c?only_with_tag=branch-db-txn#rev1.93.2.5

Signature · 3 lines
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95
#6 lore raw
In reply to Matthias Andree
Quoted 8 lines · Matthias Andree

Jens Axboe schrieb am 2004-03-22:

Quoted 4 lines

There's no such thing as atomic writes bigger than a sector really, we just pretend there is. Timing usually makes this true.

Can you explain about the timing?

Quoted 4 lines

If there is no such atomicity (except maybe in ext3fs data=journal or the upcoming reiserfs4 - isn't there?), then nobody should claim so.

Well, nobody is going to use anything except reiser4 are they?;-).....

I think that we are able to guarantee that the write is fully atomic regardless of what the block layer does, so long as the block layer respects our ordering and does not cache it where it should not.

zam, you are watching this thread about flushing the ide cache I hope....

Quoted 36 lines

If
the kernel cannot 100.00000000% guarantee the write is atomic, claiming otherwise is plain fraud and nothing else.

Some people bet their whole business/company and hence a fair deal of their belongings on a single data base, and making them believe facts that simply aren't reality is dangerous. These people will have very little understanding for sloppiness here. Linux has no obligation to be fast or reliable, but it MUST PROPERLY AND TRUTHFULLY state what it can guarantee and what it cannot guarantee.

Quoted 5 lines

For bigger atomic writes, 2.4 SUSE kernel had some nasty hack (called blk-atomic) to prevent reordering by the io scheduler to avoid partial blocks from databases.

That does not make a write atomic if the scheduled blocks are still written one at a time (and I believe tagged command queueing won't help to unroll partial writes either).

If the hardware support is missing, it is prudent to say just that and not make any bogus promises about platter inertia and "it usually works". (who says that the filter curves adjust to the decreasing platter speed and the electronics are sustained for long enough? how about write verify and remapping broken blocks?)

So we only write one hardware block size atomically, usually 512 bytes on ATA and SCSI disk drives (MO might do 2048 at a time, but why introduce complexity). That's a data point in this whole fsync() discussion.

Signature · 1 line
Hans