precise characterization of ext3 atomicity
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
I am trying to define the difference between "Atomic Reiser4" and ext3, as it seems to be a frequently asked question, and I am thinking of saying something like:
Reiser4 allows you to define a set of up to A separate arbitrary
filesystem operations (where A by default is not allowed to exceed 64) that are to be committed to disk atomically. Every individual filesystem operation is atomic without the need to specify it.
By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.
Signature · 1 line
Hans
Quoted 3 lines · Hans Reiser
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
Yes.
Quoted 4 lines
By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.
Not sure that I understand this. In data=writeback mode, metadata integrity is preserved but data writes may be lost. In data=journal and data=ordered modes the data and the metadata which refers to it are always in sync on-disk.
Quoted 16 lines · Andrew Morton
Quoted 3 lines · Hans Reiser
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
Yes.
Quoted 4 lines
By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.
Quoted 9 lines
Not sure that I understand this. In data=writeback mode, metadata integrity is preserved but data writes may be lost. In data=journal and data=ordered modes the data and the metadata which refers to it are always in sync on-disk.
Perhaps the following is correct?
By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your application's data is corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).
Signature · 1 line
Hans
Quoted 7 lines · Hans Reiser
Perhaps the following is correct?
By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal
metadata will not be corrupted even if your application's data is
corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).
Correct != comprehensible ;)
"In all journalling modes ext3 guarantees metadata consistency after a
crash. In its data=journal and data=ordered modes ext3 also guarantees that user data is consistent with metadata after a crash.
However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery"
Quoted 12 lines · Andrew Morton
Quoted 6 lines · Hans Reiser
Perhaps the following is correct?
By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your application's data is corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).Correct != comprehensible ;)
Touche!
Let's try this then:
Ext3 guarantees that its metadata will be comitted sufficiently atomically that after a crash it will be consistent with itself.
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.
Signature · 1 line
Hans
Quoted 7 lines · Hans Reiser
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single
write()spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.
And how does reiser4 do this without changing the userspace apps?
Most files are written with several write() calls, so even if each call is atomic, your entire file will not be there.
Also, ext3 could claim the same atomicity if it only updated meta-data on write() call boundaries, instead of block boundaries.
Quoted 10 lines · Mike Fedyk
Quoted 7 lines · Hans Reiser
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single
write()spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.And how does reiser4 do this without changing the userspace apps?
It won't.
[ snip ]
Quoted 5 lines
Most files are written with several
write()calls, so even if each call is atomic, your entire file will not be there.Also, ext3 could claim the same atomicity if it only updated meta-data on
write()call boundaries, instead of block boundaries.
There will be a new API to support userspace-controlled multifile transactions.
At first stab, multifile transactions will be used internally to implement extended attributes.
Now, another question is.. will the transaction API support commit() and rollback()? *grin*
(wonder about coding a simple transactional database with
shell scripts ;)
Signature · 6 lines
winden/network 1. Dado un programa, siempre tiene al menos un fallo. 2. Dadas varias lineas de codigo, siempre se pueden acortar a menos lineas. 3. Por induccion, todos los programas se pueden reducir a una linea que no funciona.
Quoted 9 lines · Mike Fedyk
Quoted 3 lines · Hans Reiser
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single
write()spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.And how does reiser4 do this without changing the userspace apps?
We don't. We just make the hovercraft, we don't force you to go over the water.....
Quoted 10 lines
Most files are written with several
write()calls, so even if each call is atomic, your entire file will not be there.Also, ext3 could claim the same atomicity if it only updated meta-data on
write()call boundaries, instead of block boundaries.
Signature · 1 line
Hans
Quoted 15 lines · Hans Reiser
Quoted 13 lines · Mike Fedyk
Quoted 9 lines · Hans Reiser
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single
write()spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.
Quoted 4 lines
Quoted 2 lines
And how does reiser4 do this without changing the userspace apps?
We don't. We just make the hovercraft, we don't force you to go over the water.....
So by default with no user space modifications, reiser4 will be atomic for each write() call, and ext3 will if it aligns withing a single page.
Is that correct?
Then you can go on to specify that you can have larger transactions if you make some changes to the userspace apps.
Quoted 21 lines · Mike Fedyk
Quoted 18 lines · Hans Reiser
Quoted 14 lines · Mike Fedyk
Quoted 5 lines · Hans Reiser
In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single
write()spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.And how does reiser4 do this without changing the userspace apps?
Quoted 9 lines
Quoted 3 lines
We don't. We just make the hovercraft, we don't force you to go over the water.....
So by default with no user space modifications, reiser4 will be atomic for each
write()call, and ext3 will if it aligns withing a single page.Is that correct?
Yes.
Quoted 7 lines
Then you can go on to specify that you can have larger transactions if you make some changes to the userspace apps.
or you are a programmer who writes code....;-) It's not that hard to write code....;-)
Signature · 1 line
Hans
Quoted 6 lines · Hans Reiser
Quoted 2 lines · Mike Fedyk
And how does reiser4 do this [export atomic ops to userspace] without changing the userspace apps?
We don't. We just make the hovercraft, we don't force you to go over the water.....
It is possible to do the same with ext3, namely exporting journal_start() and journal_stop() (or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.
If the app doesn't eventually say "end the transaction", the filesystem might wait indefinitely. You could start adding more plumbing like "if the file is closed (maybe because the process crashed), cancel the transaction", and "if the process doesn't complete the transaction in time, cancel the transaction", etc. How do you guarantee in advance that the application will be able to complete all of the operations it needs (i.e. if it runs out of space in the filesystem or something)?
I suppose at worst, the application doesn't get its multi-op atomicity guarantee, but I'm guessing that apps which use this interface depend on it working properly or they wouldn't be using it.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 14 lines · Andreas Dilger
Quoted 10 lines · Hans Reiser
Quoted 4 lines · Mike Fedyk
And how does reiser4 do this [export atomic ops to userspace] without changing the userspace apps?
We don't. We just make the hovercraft, we don't force you to go over the water.....
Quoted 7 lines
It is possible to do the same with ext3, namely exporting
journal_start()andjournal_stop()(or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.
Yup. That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.
Quoted 9 lines
If the app doesn't eventually say "end the transaction", the filesystem might wait indefinitely. You could start adding more plumbing like "if the file is closed (maybe because the process crashed), cancel the transaction", and "if the process doesn't complete the transaction in time, cancel the transaction", etc. How do you guarantee in advance that the application will be able to complete all of the operations it needs (i.e. if it runs out of space in the filesystem or something)?
We will export our space reservation infrastructure code also. We are still thinking about the right API for that. At first I had some idea that we should calculate for the user how much space would be needed, but I am getting lazier as we get closer to actually doing it, and I am thinking we can add the helpful but complex in some cases estimate_sizeof() functions later, and for now just let the user grab space, and then if they exceed it return an error, and if they both exceed it and run out of disk space return a nastier error that tells them to go cope with their mistake. Now I think it will be something that takes a 64 bit int that is the number of blocks to grab, compares it to their quota if any, and causes the sys_reiser4 to do nothing and error nicely if it can't get it.
When coding sometimes you have to be careful not to let the complex needs of the 20% prevent you from getting something that the 80% need and would be happy with to market. (Learned this one from Dave Hitz of NetApp.)
There's a lot of applications that have very simple needs in regards to atomicity, like write 3 things to 3 files as one atom. If we can address that, then later people can do their PhDs on the needs of the complex 20%.... and hopefully send some nice patches.....
Quoted 15 lines
I suppose at worst, the application doesn't get its multi-op atomicity guarantee, but I'm guessing that apps which use this interface depend on it working properly or they wouldn't be using it.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Signature · 1 line
Hans
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Andreas Dilger
It is possible to do the same with ext3, namely exporting
journal_start()andjournal_stop()(or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.
Quoted 5 lines
That's why we confine it to a (finite #defined number) set of operations within one
sys_reiser4call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supportingsys_reiser4at some point.
Ah, OK. If you are confining the atom to a single syscall, then this is a much easier problem to solve, assuming sys_reiser4() has a sufficiently rich interface to express what people want to do. It avoids all of the potential problems that could arise if you want to keep a transaction open over multiple syscalls.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 9 lines · Andreas Dilger
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Andreas Dilger
It is possible to do the same with ext3, namely exporting
journal_start()andjournal_stop()(or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.
Quoted 5 lines
Quoted 5 lines
That's why we confine it to a (finite #defined number) set of operations within one
sys_reiser4call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supportingsys_reiser4at some point.
Please rename sys_reiser4 if you want it to be a generic use syscall ;-)
-chris
Hi!
Quoted 5 lines
Yup. That's why we confine it to a (finite #defined number) set of operations within one
sys_reiser4call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supportingsys_reiser4at some point.
Well, if you want that API to be widely usable, you should invent better name than sys_reiser4 :-).
Signature · 2 lines
Pavel Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
Quoted 9 lines · Pavel Machek
Quoted 5 lines
Yup. That's why we confine it to a (finite #defined number) set of operations within one
sys_reiser4call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supportingsys_reiser4at some point.Well, if you want that API to be widely usable, you should invent better name than
sys_reiser4:-).
Like ActiveFSControll or such? ;-)
- Gábor (larta'H)
Quoted 8 lines · G?bor L?n?rt
Quoted 7 lines · Pavel Machek
Quoted 5 lines
Yup. That's why we confine it to a (finite #defined number) set of operations within one
sys_reiser4call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supportingsys_reiser4at some point.
Quoted 4 lines
Quoted 2 lines
Well, if you want that API to be widely usable, you should invent better name than
sys_reiser4:-).Like ActiveFSControll or such? ;-)
How about sys_group_journal_ops?
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
Yes.
Is that just happenstance, or does Posix or similar mandate it?
Regards,
Daniel
Quoted 9 lines · Daniel Phillips
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
Yes.
Is that just happenstance, or does Posix or similar mandate it?
Happenstance.
It's semi-trivial to do this in ext3. You'd open the file with O_ATOMIC and a write() would either be completely atomic or would return -EFOO without having written anything.
The thing which prevents this is the ranking order between journal_start() and lock_page().
It's not trivial but also not too hard to change things so that journal_start() can rank outside lock_page() - this would also offer some CPU savings.
Can't say that I'm terribly motivated about the feature though.
Quoted 16 lines · Andrew Morton
Quoted 8 lines · Daniel Phillips
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser
Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?
Yes.
Is that just happenstance, or does Posix or similar mandate it?
Happenstance.
It's semi-trivial to do this in ext3. You'd open the file with
O_ATOMICand awrite()would either be completely atomic or would return-EFOOwithout having written anything.
Quoted 8 lines
The thing which prevents this is the ranking order between
journal_start()andlock_page().It's not trivial but also not too hard to change things so that
journal_start()can rank outsidelock_page()- this would also offer some CPU savings.Can't say that I'm terribly motivated about the feature though.
I'm relieved, I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.
Regards,
Daniel
Quoted 4 lines · Daniel Phillips
I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.
Not sure what you mean.
Signature · 1 line
Hans
Quoted 6 lines · Hans Reiser
Quoted 3 lines · Daniel Phillips
I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.
Not sure what you mean.
Nothing bad. More power to you for adding a transaction interface to Reiser4, and blazing that trail. It's totally missing as a generic api at the moment, and needs a push.
Regards,
Daniel