O_DIRECT please; Sybase 12.5

At work I had to sit through a meeting where I heard the boss say "If Linux makes Sybase go through the page cache on reads, maybe we'll just have to switch to Solaris. That's a serious performance problem."
All I could say was "I expect Linux will support O_DIRECT soon, and Sybase will support that within a year."

Er, so did I promise too much? Andrea mentioned O_DIRECT recently ( http://marc.theaimsgroup.com/?l=linux-kernel&m=99253913516599&w=2,
http://lwn.net/2001/0510/bigpage.php3 )
Is it supported yet in 2.4, or is this a 2.5 thing?

And what are the chances Sybase will support that flag any time soon? I just read on news://forums.sybase.com/sybase.public.ase.linux that Sybase ASE 12.5 was released today, and a 60 day eval is downloadable for NT and Linux. I'm downloading now; it's a biggie.

It supports raw partitions, which is good; that might satisfy my boss (although the administration will be a pain, and I'm not sure whether it's really supported by Dell RAID devices). I'd prefer O_DIRECT :-(

Hope somebody can give me encouraging news.

Thanks,
Dan

Thats something you'd have to benchmark. It depends on a very large number of factors including whether the database uses mmap, the average I/O size and the like

We already support raw direct I/O to devices themselves so they should support that - if not then Oracle I believe already does.

I'll probably benchmark raw vs. non-raw I/O with Sybase ASE 12.5 on our application once we've come up to speed on basic performance issues (we're database newbies).

Haven't seen Sybase talk about O_DIRECT. Not sure we want to pony up the Sybase license fees. (I'm still in denial about databases in general, and hope I can switch to PostgreSQL at some point.)

BTW,
http://eval.veritas.com/webfiles/whitepapers/sybaseedition/sybase14_performance_paper.pdf
seems to show that raw beats O_DIRECT hands down on Solaris. Will that hold on Linux, or is your (forthcoming?) O_DIRECT higher performance than the one on Solaris?

Thanks,
Dan

Quite obviously. One of the primary things a DBA is supposed to do is ensure that the disk is accessed as *few* times as possible. What size database do you have? How much memory has the machine have? How much memory does the database have? How many engines is the database running?

We can take this off-list if you want, but disk I/O shouldn't really be an issue for any database as long as other parameters are set correctly. Sybase recommends raw devices *not* because they are faster, but because it's the only way that they (Sybase) can guarantee the data is actually written to disk (legal liability, etc.).

/Mike (Sybase DBA)

XFS supports O_DIRECT on linux, has done for a while.

Steve

Hi,

All block devices support raw IO --- the raw IO mechanism talks to the device driver through the normal kernel-internal block IO entry points.

Andrea Arcangeli has already posted patches you can try for ext2. The functionality isn't in the mainline kernel yet, though.

--Stephen

I recall hearing about a problem with the md device and raw IO. It was something about the block sizes not matching causing performance problems. Has anything been done to improve those issues?

- |Daryll

Hi,

The problem is a combination of two things. First, raw IO is always fully synchronous, so with raw IO (and O_DIRECT) you are, in effect, explicitly instructing the kernel not to do any readahead. That makes it hard to keep two disks running in parallel with soft raid if you are using small IOs, obviously.

Secondly, raw IO pins buffers in physical memory, and to avoid causing serious VM problems due to having too much unswappable memory pinned by arbitrary applications, the current raw IO driver limits the pinned chunk size to 64k. That, combined with the sequential nature of raw IO, can limit performance, certainly.

Raw IO is quite capable of running with larger chunk sizes, but we really need a kernel limiter of some description to prevent users from using this mechanism to pin massive amounts of memory for raw IO at once. There are several candidate mechanisms for that, but none in the main kernel right now.

Cheers,
Stephen

all 2.4 kernel in SuSE 7.2 ships with O_DIRECT enabled by default for ext2, just open your files with O_DIRECT as luser and there you go. Today I got in my inbox a patch from Chris Wedgwood for reiserfs, and Andrew Morton took care of ext3 O_DIRECT support (included into the ext3 patch and conditional to #ifdef KERNEL_HAS_O_DIRECT that he asked me to add to the latest o_direct patches). (you know O_DIRECT is 99% common code, so supporting new fs is almost a no brainer)

I will send the o_direct patch to Linus for 2.4 too but possibly this is 2.5 material, however I will fully support it for 2.4 too indeed as it is rock solid and you can just use it in production, same thing that everybody has to do for rawio in 2.2.

I will release a new patch soon against 2.4.7pre2 in the next aa patchkit as soon as I finished to synchronize my tree.

Andrea

correct, here the benchmarks:

	http://boudicca.tux.org/hypermail/linux-kernel/2001week17/1175.html
        http://boudicca.tux.org/hypermail/linux-kernel/2001week17/att-1175/01-directio.png

of course the huge improvement is also because of broken VM in the buffered-io case.

Andrea

Sorry, haven't looked at that yet.

ext3 journals data. That's unique and it breaks things (or rather,
things break it). It'd be trivial to support O_DIRECT in ext3's
writeback mode (metadata-only), but nobody uses that.

of generic_direct_IO() and brw_kiovec() at least.

I'll take a closer look.

-

I thought everybody uses metadata-only to avoid killing data-write performance. So I thought it was ok to at first support O_DIRECT only for metadata journaling, doing that should be a three liner as you said and that is what I expected.

brw_kiovec is called by generic_direct_IO, so yes, all you need is a private generic_direct_IO implementation to deal with the journaled data writes.

OK, thanks!

Andrea

ext3 has three modes:

data=journal

Data is journalled. Yes, this slows things down significantly.

data=ordered

The default mode and the most popular. All data is written to disk prior to a commit. Write throughput is good, and you don't have uninitialised data in your files after a crash.

data=writeback

Metadata-only. Better write throughput (in dbench, anyway),
but only metadata integrity is preserved after a crash. ie:
fsck says the fs is fine, but files can (and almost always do) contain random stuff after a crash.

Ordered data mode is really nice. It's not magical though - for example, if you reset the machine during a kernel build, a subsequent `make' will fail because you have a number of .o files which have zero length. That's the length they happened to have when the machine went down.

For ordered-data mode we need to keep track of all the buffers which are associated with a transaction's journalled metadata and ensure that they are written out before the transaction commits. That is done with a little structure which hangs off ->b_private.

Yup. metadata-only journalling is all-round much, much simpler.

FYI, that particular problem will disappear with the 2.5 Makefiles. The zero length .o files will still exist but the post-compile dependency data (.o.d) will not exist so a subsequent make kernel will rebuild the incomplete objects. This is a general workaround for incomplete kernel objects, independent of the file system type.