O_DIRECT question
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Nick Piggin
Quoted 7 lines · Aubrey
Quoted 3 lines · Nick Piggin
What you _really_ want to do is avoid large mallocs after boot, or use a CPU with an mmu. I don't think nommu linux was ever intended to be a simple drop in replacement for a normal unix kernel.
Quoted 6 lines
Quoted 3 lines
Is there a position available working on mmu CPU? Joking, :) Yes, some problems are serious on nommu linux. But I think we should try to fix them not avoid them.
Exactly, and the *real* fix is to modify userspace not to make >
PAGE_SIZEmallocs[*] if it is to be nommu friendly.
IMO it's better to go back to a 16-bit segmented system like 80286 than to artificially limit yourself to 12 bit memory chunks. Even if you don't have segments, offering a DOS or old MacOS-like memory management (allocating a fixed block on program start) is way better than "We want to cache, no matter what it costs".
If you throw away the cache, maybe you'll be slow, but if you throw away the application, you'll go backwards. If the cache is a problem, allocate one block of cache, and everybody will be happy. Maybe it's oldscool, but it works.
Quoted 2 lines
It is the kernel hacks to do things
like limit cache size that are the bandaids.
Limiting the cache is a feature, since it avoids constantly swapping out e.g. X11's keyboard mouse routines just because you opened a large picture in gimp*. Playing with the provided knobs did help for some cases, but I didn't succeed for the serious cases, and I'm not the world's dumbest computer user. Having a simple knob "don't grow larger than $num if you have to evict programs, don't go below $num2" would be THE knob any joe luser can understand and mostly DTRT or at least what you'd expect.
* I hope the vm_pps will help, but I did not yet read it's docs
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Michael Tokarev
(No, really - this load isn't entirely synthetic. It's a typical database workload - random I/O all over, on a large file. If it can, it combines several I/Os into one, by requesting more than a single block at a time, but overall it is random.)
Quoted 5 lines
My point is that you can get basically ALL THE SAME GOOD BEHAVIOUR without having all the BAD behaviour that
O_DIRECTadds.For example, just the requirement that
O_DIRECTcan never create a file mapping,
That sounds sane, but the video streaming folks will be unhappy.
Maybe you could do:
reserve_space(); (*)
do_write_odirect();
update_filesize();
and only allow reads up to the current filesize?
Off cause if you do ftruncate first and then write o_direct, the holes will need to be filled before the corresponding blocks are assigned to the file. Either you'll zero them or you can insert them into the file after the write.
Races:
against other reads: May happen in any order, to-be-written pages are
beyond filesize (inaccessible), zeroed or not yet assigned to the file.
against other writes: No bad effect, since you don't unreserve
mappings, and update_filesize won't shrink the file. You must, however, not reserve two chunks for the same location in the file unless you can handle replacing blocks of files.
open(O_WRITE) without O_DIRECT is not allowed, therefore that can't race.
against truncate: Yes, see below
(*) This would allow fadvise_size(), too, which could reduce fragmentation
(and give an early warning on full disks) without forcing e.g. fat to zero all blocks. OTOH, fadvise_size() would allow users to reserve the complete disk space without his filesizes reflecting this.
Quoted 1 line
and can never interact with ftruncate
ACK, r/w semaphore, read={r,w}_odirect, write=ftruncate?
Quoted 4 lines
would actually make
O_DIRECTa lot more palatable to me. Together with just the requirement that anO_DIRECTopen would literally disallow any non-O_DIRECT accesses, and flush the page cache entirely, would make all the aliases go away.
That's probably the best semantics.
Maybe you should allow O_READ for the backup people, maybe forcing O_DIRECT|O_ALLOWDOUBLEBUFFER (doing the extra copy in the kernel).
Quoted 3 lines
At that point,
O_DIRECTwould be a way of saying "we're going to do uncached accesses to this pre-allocated file". Which is a half-way sensible thing to do.
And I'd bet nobody would notice these changes unless they try inherently stupid things.
Quoted 3 lines
But what
O_DIRECTdoes right now is _not_ really sensible, and theO_DIRECTpropeller-heads seem to have some problem even admitting that there _is_ a problem, because they don't care.
It's a hammer - having it will make anything look like a nail, and there is nothing wrong with hammering a nail!!! .-)
Quoted 3 lines
A lot of DB people seem to simply not care about security or anything else.anything else. I'm trying to tell you that quoting numbers is pointless, when simply the CORRECTNESS of
O_DIRECTis very much in doubt.
The only thing you'll need for a correct database behaviour is:
If one process has completed it's write and the next process opens that file, it must read the current contents.
Races with normal reads and writes, races with truncate - don't do that then. You wouldn't expect "cat somefile > database.dat" on a running db to be a good thing, too, no matter if o_direct is used or not.
Signature · 4 lines
Funny quotes: 3. On the other hand, you have different fingers. Friß, Spammer: [email protected]
Quoted 4 lines · Bodo Eggert
(*) This would allow
fadvise_size(), too, which could reduce fragmentation
(and give an early warning on full disks) without forcing e.g. fat to zero all blocks. OTOH,fadvise_size()would allow users to reserve the complete disk space without his filesizes reflecting this.
Please clarify how this would interact with quota, and why it wouldn't allow someone to run me out of disk.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 9 lines · Bill Davidsen
Quoted 4 lines · Bodo Eggert
(*) This would allow
fadvise_size(), too, which could reduce fragmentation
(and give an early warning on full disks) without forcing e.g. fat to zero all blocks. OTOH,fadvise_size()would allow users to reserve the complete disk space without his filesizes reflecting this.Please clarify how this would interact with quota, and why it wouldn't allow someone to run me out of disk.
I fell into the "write-will-never-fail"-pit. Therefore I have to talk about the original purpose, write with O_DIRECT, too.
- Reserved blocks should be taken out of the quota, since they are about to be written right now. If you emptied your quota doing this, it's your fault. It it was the group's quota, just run fast enough.-)
- If one write failed that extended the reserved range, the reserved area should be shrunk again. Obviously you'll need something clever here.
* You can't shrink carelessly while there are O_DIRECT writes.
* You can't just try to grab the semaphore[0] for writing, this will deadlock with other write()s.
* If you drop the read lock, it will work out, because you aren't writing anymore, and if you get the write lock, there won't be anybody else writing. Therefore you can clear the reservation for the not- written blocks. You may unreserve blocks that should stay reserved, but that won't harm much. At worst, you'll get fragmentation, loss of speed and an aborted (because of no free space) write command. Document this, it's a feature.-)
- If you fadvise_size on a non-quota-disk, you can possibly reserve it completely, without being the easy-to-spot offender. You can do the same by actually writing these files, keeping them open and unlinking them. The new quality is: You can't just look at the file sizes in /proc in order to spot the offender. However, if you reflect the reserved blocks in the used-blocks-field of struct stat, du will work as expected and the BOFH will know whom to LART.
BTW: If the fs supports holes, using du would be the right thing to do anyway.
BTW2: I don't know if reserving without actually assigning blocks is supported or easy to support at all. These reservations are the result of "These blocks are not yet written, therefore they contain possibly secret data that would leak on failed writes, therefore they may not be actually assigned to the file before write finishes. They may not be on the free list either. And hey, if we support pre-reserving blocks to the file, we may additionally use it for fadvise_size. I'll mention that briefly."
[0] r/w semaphore, read={r,w}_odirect, write=ftruncate
Signature · 3 lines
Fun things to slip into your budget Paradigm pro-activator (a whole pack) (you mean beer?)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Bill Davidsen
My point is, that there is code to handle sparse data now, without
O_DIRECTinvolved, and ifO_DIRECTbypasses that, it's not a problem with the idea ofO_DIRECT, the kernel has a security problem.
The idea of O_DIRECT is to bypass the pagecache, and the pagecache is what provides the security against reading someone else's data using sparse files or partial-block-IO.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Helge Hafting
Quoted 3 lines · Michael Tokarev
But seriously - what about just disallowing non-O_DIRECT opens together with
O_DIRECTones ?Please do not create a new local DOS attack. I open some important file, say
/etc/resolv.confwithO_DIRECTand just sit on the open handle. Now nobody else can open that file because it is "busy" withO_DIRECT?
Suspend O_DIRECT access while non-O_DIRECT-fds are open, fdatasync on close?
Signature · 4 lines
"Unix policy is to not stop root from doing stupid things because that would also stop him from doing clever things." - Andi Kleen "It's such a fine line between stupid and clever" - Derek Smalls
Quoted 6 lines · Bodo Eggert
Quoted 5 lines · Helge Hafting
Quoted 3 lines · Michael Tokarev
But seriously - what about just disallowing non-O_DIRECT opens together with
O_DIRECTones ?
Quoted 7 lines
Quoted 5 lines
Please do not create a new local DOS attack. I open some important file, say
/etc/resolv.confwithO_DIRECTand just sit on the open handle. Now nobody else can open that file because it is "busy" withO_DIRECT?Suspend
O_DIRECTaccess while non-O_DIRECT-fds are open, fdatasync on close?
.. then any user can impact the operation, performance and reliability of the database application of another user... sounds like plugging one hole by making a bigger hole ;)
Quoted 13 lines · Arjan van de Ven
Quoted 12 lines · Bodo Eggert
Quoted 10 lines · Helge Hafting
Quoted 3 lines · Michael Tokarev
But seriously - what about just disallowing non-O_DIRECT opens together with
O_DIRECTones ?Please do not create a new local DOS attack. I open some important file, say
/etc/resolv.confwithO_DIRECTand just sit on the open handle. Now nobody else can open that file because it is "busy" withO_DIRECT?
Quoted 5 lines
Quoted 1 line
Suspend
O_DIRECTaccess while non-O_DIRECT-fds are open, fdatasync on close?.. then any user can impact the operation, performance and reliability of the database application of another user... sounds like plugging one hole by making a bigger hole ;)
Don't allow other users to access your raw database files then, and if backup kicks in, pausing the database would DTRT for integrety of the backup. For other applications, paused O_DIRECT may very well be a problem, but I can't think of one right now.
Signature · 1 line
Logic: The art of being wrong with confidence...
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Denis Vlasenko
Quoted 6 lines · Bill Davidsen
Quoted 5 lines · Denis Vlasenko
Quoted 2 lines · Michael Tokarev
But even single-threaded I/O but in large quantities benefits from
O_DIRECTsignificantly, and I pointed this out before.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Which shouldn't be true. There is no fundamental reason why ordinary writes should be slower than
O_DIRECT.Other than the copy to buffer taking CPU and memory resources.
It is not required by any standard that I know. Kernel can be smarter and avoid that if it can.
The kernel can also solve the halting problem if it can.
Do you really think an entropy estamination code on all access patterns in the system will be free as in beer, or be able to predict the access pattern of random applications?
Signature · 4 lines
Top 100 things you don't want the sysadmin to say: 86. What do you mean that wasn't a copy? Friß, Spammer: [email protected] [email protected]
Quoted 13 lines · Bodo Eggert
Quoted 12 lines · Denis Vlasenko
Quoted 10 lines · Bill Davidsen
Quoted 8 lines · Denis Vlasenko
Quoted 2 lines · Michael Tokarev
But even single-threaded I/O but in large quantities benefits from
O_DIRECTsignificantly, and I pointed this out before.Which shouldn't be true. There is no fundamental reason why ordinary writes should be slower than
O_DIRECT.Other than the copy to buffer taking CPU and memory resources.
Quoted 7 lines
Quoted 2 lines
It is not required by any standard that I know. Kernel can be smarter and avoid that if it can.
The kernel can also solve the halting problem if it can.
Do you really think an entropy estamination code on all access patterns in the system will be free as in beer,
Actually I think we need this heuristic:
if (opened_with_O_STREAM && buffer_is_aligned
&& io_size_is_a_multiple_of_sectorsize) do_IO_directly_to_user_buffer_without_memcpy
is not *that* compilcated.
I think that we can get rid of O_DIRECT peculiar requirements "you *must* not cache me" + "you *must* write me directly to bare metal" by replacing it with O_STREAM ("*advice* to not cache me") + O_SYNC ("write() should return only when data is written to storage, not sooner").
Why?
Because these O_DIRECT "musts" are rather unusual and overkill. Apps should not have that much control over what kernel does internally; and also O_DIRECT was mixing shampoo and conditioner on one bottle (no-cache and sync writes) - bad API.
--
vda
Quoted 31 lines · Denis Vlasenko
Quoted 9 lines · Bodo Eggert
Quoted 8 lines · Denis Vlasenko
Quoted 7 lines · Bill Davidsen
Quoted 6 lines · Denis Vlasenko
Quoted 2 lines · Michael Tokarev
But even single-threaded I/O but in large quantities benefits from
O_DIRECTsignificantly, and I pointed this out before.Which shouldn't be true. There is no fundamental reason why ordinary writes should be slower than
O_DIRECT.Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Other than the copy to buffer taking CPU and memory resources.
It is not required by any standard that I know. Kernel can be smarter and avoid that if it can.
The kernel can also solve the halting problem if it can.
Do you really think an entropy estamination code on all access patterns in the system will be free as in beer,
Actually I think we need this heuristic:
if (
opened_with_O_STREAM&&buffer_is_aligned&& io_size_is_a_multiple_of_sectorsize) do_IO_directly_to_user_buffer_without_memcpyis not *that* compilcated.
I think that we can get rid of
O_DIRECTpeculiar requirements "you *must* not cache me" + "you *must* write me directly to bare metal" by replacing it withO_STREAM("*advice* to not cache me") +O_SYNC("write()should return only when data is written to storage, not sooner").
Quoted 6 lines
Why?
Because these
O_DIRECT"musts" are rather unusual and overkill. Apps should not have that much control over what kernel does internally; and alsoO_DIRECTwas mixing shampoo and conditioner on one bottle (no-cache and sync writes) - bad API.
What a shame that other operating systems can manage to really support O_DIRECT, and that major application software can use this api to write portable code that works even on Windows.
You overlooked the problem that applications using this api assume that reads are on bare metal as well, how do you address the case where thread A does a write, thread B does a read? If you give thread B data from a buffer and it then does a write to another file (which completes before the write from thread A), and then the system crashes, you have just put the files out of sync. So you may have to block all i/o for all threads of the application to be sure that doesn't happen. Or introduce some complex way to assure that all writes are physically done in order... that sounds like a lock infested mess to me, assuming that you could ever do it right.
Oracle has their own version of Linux now, do you think that they would fork the application or the kernel?
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Quoted 43 lines · Bill Davidsen
Quoted 31 lines · Denis Vlasenko
Quoted 9 lines · Bodo Eggert
Quoted 8 lines · Denis Vlasenko
Quoted 7 lines · Bill Davidsen
Quoted 6 lines · Denis Vlasenko
Quoted 2 lines · Michael Tokarev
But even single-threaded I/O but in large quantities benefits from
O_DIRECTsignificantly, and I pointed this out before.Which shouldn't be true. There is no fundamental reason why ordinary writes should be slower than
O_DIRECT.Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Other than the copy to buffer taking CPU and memory resources.
It is not required by any standard that I know. Kernel can be smarter and avoid that if it can.
The kernel can also solve the halting problem if it can.
Do you really think an entropy estamination code on all access patterns in the system will be free as in beer,
Actually I think we need this heuristic:
if (
opened_with_O_STREAM&&buffer_is_aligned&& io_size_is_a_multiple_of_sectorsize) do_IO_directly_to_user_buffer_without_memcpyis not *that* compilcated.
I think that we can get rid of
O_DIRECTpeculiar requirements "you *must* not cache me" + "you *must* write me directly to bare metal" by replacing it withO_STREAM("*advice* to not cache me") +O_SYNC("write()should return only when data is written to storage, not sooner").Quoted 6 lines
Why?
Because these
O_DIRECT"musts" are rather unusual and overkill. Apps should not have that much control over what kernel does internally; and alsoO_DIRECTwas mixing shampoo and conditioner on one bottle (no-cache and sync writes) - bad API.What a shame that other operating systems can manage to really support
O_DIRECT, and that major application software can use this api to write portable code that works even on Windows.
Quoted 6 lines
You overlooked the problem that applications using this api assume that reads are on bare metal as well, how do you address the case where thread A does a write, thread B does a read? If you give thread B data from a buffer and it then does a write to another file (which completes before the write from thread A), and then the system crashes, you have just put the files out of sync.
Applications which syncronize their data integrity by keeping data on hard drive and relying on "read goes to bare metal, so it can't see written data before it gets written to bare metal". Wow, this is slow. Are you talking about this scenario:
Bad:
fd = open(..., O_SYNC);
fork()
write(fd, buf); [1]
.... read(fd, buf2); [starts after write 1 started]
.... write(somewhere_else, buf2);
.... (write returns)
.... <---- crash point
(write returns)This will be *very* slow - if you use O_DIRECT and do what is depicted above, you write data, then you read it back, whic is slow. Why do you want that? Isn't it
much faster to just wait for write to complete, and allow read to fetch (potentially) cached data?
Better:
fd = open(..., O_SYNC);
fork()
write(fd, buf); [1]
.... (wait for write to finish)
....
....
.... <---- crash point
(write returns)
.... read(fd, buf2); [starts after write 1 started]
.... write(somewhere_else, buf2);
.... (write returns)Quoted 2 lines
So you may have to block all i/o for all
threads of the application to be sure that doesn't happen.
Not all, only related i/o.
--
vda