Skip to contentSkip to conversation

precise characterization of ext3 atomicity

DISCUSSION Hans Reiser 9 participants 22 messages Sep 4, 2003 – Sep 9, 2003

lore
Share & tools
lore
#0 lore raw

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

I am trying to define the difference between "Atomic Reiser4" and ext3, as it seems to be a frequently asked question, and I am thinking of saying something like:

Reiser4 allows you to define a set of up to A separate arbitrary
filesystem operations (where A by default is not allowed to exceed 64) that are to be committed to disk atomically. Every individual filesystem operation is atomic without the need to specify it.

By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.

Signature · 1 line
Hans
#1 lore raw
Quoted 3 lines · Hans Reiser

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

Yes.

Quoted 4 lines

By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.

Not sure that I understand this. In data=writeback mode, metadata integrity is preserved but data writes may be lost. In data=journal and data=ordered modes the data and the metadata which refers to it are always in sync on-disk.

#2 lore raw
Quoted 16 lines · Andrew Morton
Quoted 3 lines · Hans Reiser

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

Yes.

Quoted 4 lines

By contrast, ext3 only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.

Quoted 9 lines

Not sure that I understand this. In data=writeback mode, metadata integrity is preserved but data writes may be lost. In data=journal and data=ordered modes the data and the metadata which refers to it are always in sync on-disk.

Perhaps the following is correct?

By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your application's data is corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).

Signature · 1 line
Hans
#3 lore raw
Quoted 7 lines · Hans Reiser

Perhaps the following is correct?

By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal
metadata will not be corrupted even if your application's data is
corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).

Correct != comprehensible ;)

"In all journalling modes ext3 guarantees metadata consistency after a
crash. In its data=journal and data=ordered modes ext3 also guarantees that user data is consistent with metadata after a crash.

However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery"

#4 lore raw
Quoted 12 lines · Andrew Morton
Quoted 6 lines · Hans Reiser

Perhaps the following is correct?

By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write
that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your application's data is corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).

Correct != comprehensible ;)

Touche!

Let's try this then:

Ext3 guarantees that its metadata will be comitted sufficiently atomically that after a crash it will be consistent with itself.

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

Signature · 1 line
Hans
#5 lore raw
Quoted 7 lines · Hans Reiser

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

And how does reiser4 do this without changing the userspace apps?

Most files are written with several write() calls, so even if each call is atomic, your entire file will not be there.

Also, ext3 could claim the same atomicity if it only updated meta-data on write() call boundaries, instead of block boundaries.

#6 lore raw
Quoted 10 lines · Mike Fedyk
Quoted 7 lines · Hans Reiser

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

And how does reiser4 do this without changing the userspace apps?

It won't.

[ snip ]

Quoted 5 lines

Most files are written with several write() calls, so even if each call is atomic, your entire file will not be there.

Also, ext3 could claim the same atomicity if it only updated meta-data on write() call boundaries, instead of block boundaries.

There will be a new API to support userspace-controlled multifile transactions.

At first stab, multifile transactions will be used internally to implement extended attributes.

Now, another question is.. will the transaction API support commit() and rollback()? *grin*

(wonder about coding a simple transactional database with
shell scripts ;)

Signature · 6 lines
winden/network

1. Dado un programa, siempre tiene al menos un fallo.
2. Dadas varias lineas de codigo, siempre se pueden acortar a menos lineas.
3. Por induccion, todos los programas se pueden
   reducir a una linea que no funciona.
#7 lore raw
In reply to Mike Fedyk
Quoted 9 lines · Mike Fedyk
Quoted 3 lines · Hans Reiser

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

And how does reiser4 do this without changing the userspace apps?

We don't. We just make the hovercraft, we don't force you to go over the water.....

Quoted 10 lines

Most files are written with several write() calls, so even if each call is atomic, your entire file will not be there.

Also, ext3 could claim the same atomicity if it only updated meta-data on write() call boundaries, instead of block boundaries.

Signature · 1 line
Hans
#8 lore raw
Quoted 15 lines · Hans Reiser
Quoted 13 lines · Mike Fedyk
Quoted 9 lines · Hans Reiser

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

Quoted 4 lines
Quoted 2 lines

And how does reiser4 do this without changing the userspace apps?

We don't. We just make the hovercraft, we don't force you to go over the water.....

So by default with no user space modifications, reiser4 will be atomic for each write() call, and ext3 will if it aligns withing a single page.

Is that correct?

Then you can go on to specify that you can have larger transactions if you make some changes to the userspace apps.

#9 lore raw
Quoted 21 lines · Mike Fedyk
Quoted 18 lines · Hans Reiser
Quoted 14 lines · Mike Fedyk
Quoted 5 lines · Hans Reiser

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to. However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes). If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.

And how does reiser4 do this without changing the userspace apps?

Quoted 9 lines
Quoted 3 lines

We don't. We just make the hovercraft, we don't force you to go over the water.....

So by default with no user space modifications, reiser4 will be atomic for each write() call, and ext3 will if it aligns withing a single page.

Is that correct?

Yes.

Quoted 7 lines

Then you can go on to specify that you can have larger transactions if you make some changes to the userspace apps.

or you are a programmer who writes code....;-) It's not that hard to write code....;-)

Signature · 1 line
Hans
#10 lore raw
In reply to Hans Reiser
Quoted 6 lines · Hans Reiser
Quoted 2 lines · Mike Fedyk

And how does reiser4 do this [export atomic ops to userspace] without changing the userspace apps?

We don't. We just make the hovercraft, we don't force you to go over the water.....

It is possible to do the same with ext3, namely exporting journal_start() and journal_stop() (or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.

If the app doesn't eventually say "end the transaction", the filesystem might wait indefinitely. You could start adding more plumbing like "if the file is closed (maybe because the process crashed), cancel the transaction", and "if the process doesn't complete the transaction in time, cancel the transaction", etc. How do you guarantee in advance that the application will be able to complete all of the operations it needs (i.e. if it runs out of space in the filesystem or something)?

I suppose at worst, the application doesn't get its multi-op atomicity guarantee, but I'm guessing that apps which use this interface depend on it working properly or they wouldn't be using it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#11 lore raw
Quoted 14 lines · Andreas Dilger
Quoted 10 lines · Hans Reiser
Quoted 4 lines · Mike Fedyk

And how does reiser4 do this [export atomic ops to userspace] without changing the userspace apps?

We don't. We just make the hovercraft, we don't force you to go over the water.....

Quoted 7 lines

It is possible to do the same with ext3, namely exporting journal_start() and journal_stop() (or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.

Yup. That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Quoted 9 lines

If the app doesn't eventually say "end the transaction", the filesystem might wait indefinitely. You could start adding more plumbing like "if the file is closed (maybe because the process crashed), cancel the transaction", and "if the process doesn't complete the transaction in time, cancel the transaction", etc. How do you guarantee in advance that the application will be able to complete all of the operations it needs (i.e. if it runs out of space in the filesystem or something)?

We will export our space reservation infrastructure code also. We are still thinking about the right API for that. At first I had some idea that we should calculate for the user how much space would be needed, but I am getting lazier as we get closer to actually doing it, and I am thinking we can add the helpful but complex in some cases estimate_sizeof() functions later, and for now just let the user grab space, and then if they exceed it return an error, and if they both exceed it and run out of disk space return a nastier error that tells them to go cope with their mistake. Now I think it will be something that takes a 64 bit int that is the number of blocks to grab, compares it to their quota if any, and causes the sys_reiser4 to do nothing and error nicely if it can't get it.

When coding sometimes you have to be careful not to let the complex needs of the 20% prevent you from getting something that the 80% need and would be happy with to market. (Learned this one from Dave Hitz of NetApp.)

There's a lot of applications that have very simple needs in regards to atomicity, like write 3 things to 3 files as one atom. If we can address that, then later people can do their PhDs on the needs of the complex 20%.... and hopefully send some nice patches.....

Quoted 15 lines

I suppose at worst, the application doesn't get its multi-op atomicity guarantee, but I'm guessing that apps which use this interface depend on it working properly or they wouldn't be using it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Signature · 1 line
Hans
#12 lore raw
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Andreas Dilger

It is possible to do the same with ext3, namely exporting journal_start() and journal_stop() (or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.

Quoted 5 lines

That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Ah, OK. If you are confining the atom to a single syscall, then this is a much easier problem to solve, assuming sys_reiser4() has a sufficiently rich interface to express what people want to do. It avoids all of the potential problems that could arise if you want to keep a transaction open over multiple syscalls.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#13 lore raw
Quoted 9 lines · Andreas Dilger
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Andreas Dilger

It is possible to do the same with ext3, namely exporting journal_start() and journal_stop() (or some interface to them) to userspace so the application can start a transaction for multiple operations. We had discussed this in the past, but decided not to do so because user applications can screw up in so many ways, and if an application uses these interfaces it is possible to deadlock the entire filesystem if the application isn't well behaved.

Quoted 5 lines
Quoted 5 lines

That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Please rename sys_reiser4 if you want it to be a generic use syscall ;-)

-chris

#14 lore raw
In reply to Hans Reiser

Hi!

Quoted 5 lines

Yup. That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Well, if you want that API to be widely usable, you should invent better name than sys_reiser4 :-).

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#15 lore raw
Quoted 9 lines · Pavel Machek
Quoted 5 lines

Yup. That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Well, if you want that API to be widely usable, you should invent better name than sys_reiser4 :-).

Like ActiveFSControll or such? ;-)

- Gábor (larta'H)

#16 lore raw
Quoted 8 lines · G?bor L?n?rt
Quoted 7 lines · Pavel Machek
Quoted 5 lines

Yup. That's why we confine it to a (finite #defined number) set of operations within one sys_reiser4 call. At some point we will allow trusted user space processes to span multiple system calls (mail server applicances, database appliances, etc., might find this useful). You might consider supporting sys_reiser4 at some point.

Quoted 4 lines
Quoted 2 lines

Well, if you want that API to be widely usable, you should invent better name than sys_reiser4 :-).

Like ActiveFSControll or such? ;-)

How about sys_group_journal_ops?

#17 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

Yes.

Is that just happenstance, or does Posix or similar mandate it?

Regards,

Daniel

#18 lore raw
Quoted 9 lines · Daniel Phillips
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

Yes.

Is that just happenstance, or does Posix or similar mandate it?

Happenstance.

It's semi-trivial to do this in ext3. You'd open the file with O_ATOMIC and a write() would either be completely atomic or would return -EFOO without having written anything.

The thing which prevents this is the ranking order between journal_start() and lock_page().

It's not trivial but also not too hard to change things so that journal_start() can rank outside lock_page() - this would also offer some CPU savings.

Can't say that I'm terribly motivated about the feature though.

#19 lore raw
Quoted 16 lines · Andrew Morton
Quoted 8 lines · Daniel Phillips
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Hans Reiser

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

Yes.

Is that just happenstance, or does Posix or similar mandate it?

Happenstance.

It's semi-trivial to do this in ext3. You'd open the file with O_ATOMIC and a write() would either be completely atomic or would return -EFOO without having written anything.

Quoted 8 lines

The thing which prevents this is the ranking order between journal_start() and lock_page().

It's not trivial but also not too hard to change things so that journal_start() can rank outside lock_page() - this would also offer some CPU savings.

Can't say that I'm terribly motivated about the feature though.

I'm relieved, I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.

Regards,

Daniel

#20 lore raw
Quoted 4 lines · Daniel Phillips

I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.

Not sure what you mean.

Signature · 1 line
Hans
#21 lore raw
Quoted 6 lines · Hans Reiser
Quoted 3 lines · Daniel Phillips

I have always thought that some higher level synchronization is required for simultaneous writes. So Hans might as well tell his fans that Ext3 makes no official guarantee, and neither does Linux.

Not sure what you mean.

Nothing bad. More power to you for adding a transaction interface to Reiser4, and blazing that trail. It's totally missing as a generic api at the moment, and needs a push.

Regards,

Daniel