ReiserFS patch for updating ctimes of renamed files
Hi,
I ran into some trouble trying to do incremental backups with GNU tar (using --listed-incremental) where renaming a file in between backups would cause the file to disappear upon restoration. When investigating the issue I discovered that this doesn't happen on ext2, ext3, and tmpfs filesystems but only on ReiserFS filesystems. I also noticed that for example ext3 updates the affected file's ctime upon rename whereas ReiserFS doesn't, so I'm thinking this causes tar to believe that the file existed before the first backup was taking under the new name, and as a result it doesn't back it up during the second backup. So I believe ReiserFS needs to update ctimes for renamed files in order for incremental GNU tar backups to work reliably.
I made some changes to the reiserfs_rename function that I *think* should fix the problem. However, I don't know much about ReiserFS's internals, and I haven't been able to test them out to see if things work now since I can't afford to deal with potential FS corruption with my current Linux box.
I included a patch below against the 2.4.22 kernel with my changes. Would somebody mind taking a look at this to see if I did things right here (and perhaps wouldn't mind testing it out either)? If it works then I (and I'm sure others who've experienced the same problem) would like to see the changes applied to the next 2.4.x (and 2.6.x?) release.
Thanks a lot.
Alex
reiserfs/namei.c Sun Oct 12 00:39:05 2003 renamed from reiserfs/namei.c.orig Mon Aug 25 06:44:43 2003 +2 -0
@@ -1207,6 +1207,8 @@ journal_mark_dirty (&th, old_dir->i_sb, old_de.de_bh); old_dir->i_ctime = old_dir->i_mtime = CURRENT_TIME; new_dir->i_ctime = new_dir->i_mtime = CURRENT_TIME; + old_inode->i_ctime = CURRENT_TIME; + reiserfs_update_sd (&th, old_inode); if (new_dentry_inode) { // adjust link number of the victim
Quoted 19 lines · Alex Adriaanse
Hi,
I ran into some trouble trying to do incremental backups with GNU tar (using --listed-incremental) where renaming a file in between backups would cause the file to disappear upon restoration. When investigating the issue I discovered that this doesn't happen on ext2, ext3, and tmpfs filesystems but only on ReiserFS filesystems. I also noticed that for example ext3 updates the affected file's ctime upon rename whereas ReiserFS doesn't, so I'm thinking this causes tar to believe that the file existed before the first backup was taking under the new name, and as a result it doesn't back it up during the second backup. So I believe ReiserFS needs to update ctimes for renamed files in order for incremental GNU tar backups to work reliably.
I made some changes to the
reiserfs_renamefunction that I *think* should fix the problem. However, I don't know much about ReiserFS's internals, and I haven't been able to test them out to see if things work now since I can't afford to deal with potential FS corruption with my current Linux box.
Quoted 5 lines
I included a patch below against the 2.4.22 kernel with my changes. Would somebody mind taking a look at this to see if I did things right here (and perhaps wouldn't mind testing it out either)? If it works then I (and I'm sure others who've experienced the same problem) would like to see the changes applied to the next 2.4.x (and 2.6.x?) release.
Hmm. I'm conflicted.
rename(2) manpage:
Any other hard links to the file (as created using link(2)) are unaffected.
A change to ctime would affect the other links.
stat(2) manpage:
The field st_ctime is changed by writing or by setting inode information (i.e., owner, group, link count, mode, etc.).
I am not aware of any field in the inode structure that must be changed by an atomic rename. Per documentation the only reason rename should update st_ctime is if it does a link+unlink sequence which would alter st_nlink briefly.
On the other hand it does seem to me there ought to be some record that something about the inode changed. st_ctime would be the only appropriate indicator.
rename() SUSv3:
Some implementations mark for update the st_ctime field of renamed files and some do not. Applications which make use of the st_ctime field may behave differently with respect to renamed files unless they are designed to allow for either behavior.
So reiserfs is on this point definitely standards conformant already. A change could at best be seen as an enhancement.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 59 lines · jw schultz
Quoted 19 lines · Alex Adriaanse
Hi,
I ran into some trouble trying to do incremental backups with GNU tar (using --listed-incremental) where renaming a file in between backups would cause the file to disappear upon restoration. When investigating the issue I discovered that this doesn't happen on ext2, ext3, and tmpfs filesystems but only on ReiserFS filesystems. I also noticed that for example ext3 updates the affected file's ctime upon rename whereas ReiserFS doesn't, so I'm thinking this causes tar to believe that the file existed before the first backup was taking under the new name, and as a result it doesn't back it up during the second backup. So I believe ReiserFS needs to update ctimes for renamed files in order for incremental GNU tar backups to work reliably.
I made some changes to the
reiserfs_renamefunction that I *think* should fix the problem. However, I don't know much about ReiserFS's internals, and I haven't been able to test them out to see if things work now since I can't afford to deal with potential FS corruption with my current Linux box.Quoted 7 lines
I included a patch below against the 2.4.22 kernel with my changes. Would somebody mind taking a look at this to see if I did things right here (and perhaps wouldn't mind testing it out either)? If it works then I (and I'm sure others who've experienced the same problem) would like to see the changes applied to the next 2.4.x (and 2.6.x?) release.
Hmm. I'm conflicted.
rename(2) manpage:
Any other hard links to the file (as created using link(2)) are unaffected.A change to ctime would affect the other links.
stat(2) manpage:
The fieldst_ctimeis changed by writing or by setting inode information (i.e., owner, group, link count, mode, etc.).I am not aware of any field in the inode structure that must be changed by an atomic rename. Per documentation the only reason rename should update
st_ctimeis if it does a link+unlink sequence which would alterst_nlinkbriefly.On the other hand it does seem to me there ought to be some record that something about the inode changed.
st_ctimewould be the only appropriate indicator.
rename()SUSv3:
Some implementations mark for update thest_ctimefield of renamed files and some do not. Applications which make use of thest_ctimefield may behave differently with respect to renamed files unless they are designed to allow for either behavior.
Quoted 8 lines
So reiserfs is on this point definitely standards conformant already. A change could at best be seen as an enhancement.
thanks Mr. Schultz, you saved us a lot of work in reviewing this issue.
In theory it is cleaner and purer to do it the way we did. In practice, Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
Signature · 1 line
Hans
A message that is not in the archive (sent off-list or not yet received): [email protected]
Alex, are you convinced by jw? (I think I am.) Would you be willing to submit a patch for tar instead?
Hans
Quoted 23 lines · jw schultz
On Mon, Oct 13, 2003 at 09:49:20AM +0400, Hans Reiser wrote:
In theory it is cleaner and purer to do it the way we did. In practice,
Quoted 3 lines
Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
Updating ctime does seem messy and a bit irrelevant for the atomic rename. You are modifying the directories not the fricken file. This isn't DOS! But it would seem he does indeed have an issue although i'm not sure what. I've never used the listed-incremental option of tar and since the manpage is incomplete <rant deleted> i don't know what it actually does. However, i have found the use of ctime to be terribly unreliable for file management and given what the standards have to say on the issue it sounds like tar is being abused or has a bug.
Signature · 1 line
Hans
Hans,
Yes, I agree with J.W. However, I also think that Andrew has a good point in that the behavior across Linux filesystems (ReiserFS, ext2, ext3, minix, etc.) should be consistent. Either they should all update ctime during renames, or none of them should.
Anyway, I'll try to work with the GNU tar maintainer to get this problem in tar fixed. It'll probably be a lot harder to fix in tar than to have ReiserFS update ctimes since it'll require major changes in
the --listed-incremental snapshot files. However, if you don't think it's a good idea to make these changes to ReiserFS then we'll just work on fixing up tar.
Thanks,
Alex
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Hans Reiser
Sent: Monday, October 13, 2003 3:46 AM
To: Alex Adriaanse
Cc: jw schultz; Linux Kernel Mailing List; [email protected]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Alex, are you convinced by jw? (I think I am.) Would you be willing to submit a patch for tar instead?
Hans
Quoted 11 lines · jw schultz
On Mon, Oct 13, 2003 at 09:49:20AM +0400, Hans Reiser wrote:
In theory it is cleaner and purer to do it the way we did. In practice,
Quoted 5 lines
Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
Quoted 14 lines
Updating ctime does seem messy and a bit irrelevant for the atomic rename. You are modifying the directories not the fricken file. This isn't DOS! But it would seem he does indeed have an issue although i'm not sure what. I've never used the listed-incremental option of tar and since the manpage is incomplete <rant deleted> i don't know what it actually does. However, i have found the use of ctime to be terribly unreliable for file management and given what the standards have to say on the issue it sounds like tar is being abused or has a bug.
--
Hans
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Let's see what Andrew says after he reads J.W.'s reasoning. I agree that reiserfs should do the same thing as the other filesystems in Linux, but J.W. seems to be right that they are doing the wrong thing.
Hans
Quoted 82 lines · Alex Adriaanse
Hans,
Yes, I agree with J.W. However, I also think that Andrew has a good point in that the behavior across Linux filesystems (ReiserFS, ext2, ext3, minix, etc.) should be consistent. Either they should all update ctime during renames, or none of them should.
Anyway, I'll try to work with the GNU tar maintainer to get this problem in tar fixed. It'll probably be a lot harder to fix in tar than to have ReiserFS update ctimes since it'll require major changes in
the --listed-incremental snapshot files. However, if you don't think it's a good idea to make these changes to ReiserFS then we'll just work on fixing up tar.Thanks,
Alex
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Hans Reiser
Sent: Monday, October 13, 2003 3:46 AM
To: Alex Adriaanse
Cc: jw schultz; Linux Kernel Mailing List; [email protected]
Subject: Re: ReiserFS patch for updating ctimes of renamed filesAlex, are you convinced by jw? (I think I am.) Would you be willing to submit a patch for tar instead?
Hans
Quoted 14 lines · jw schultz
On Mon, Oct 13, 2003 at 09:49:20AM +0400, Hans Reiser wrote:
In theory it is cleaner and purer to do it the way we did. In practice,
Quoted 7 lines
Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
Quoted 16 lines
Updating ctime does seem messy and a bit irrelevant for the atomic rename. You are modifying the directories not the fricken file. This isn't DOS! But it would seem he does indeed have an issue although i'm not sure what. I've never used the listed-incremental option of tar and since the manpage is incomplete <rant deleted> i don't know what it actually does. However, i have found the use of ctime to be terribly unreliable for file management and given what the standards have to say on the issue it sounds like tar is being abused or has a bug.
--
Hans-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
Hans
Quoted 3 lines · Hans Reiser
Let's see what Andrew says after he reads J.W.'s reasoning. I agree that reiserfs should do the same thing as the other filesystems in Linux, but J.W. seems to be right that they are doing the wrong thing.
Whoa there. I am not saying that the other filesystems are wrong. All i am saying as i can see no reason why, as defined, the ctime of a file should be updated when nothing else in the inode structure as been updated.
Of course if i had designed it in the first place with the filesystem semantics that we have now there might be no rename system call. Renames would be done by link(oldpath, newpath); unlink(oldpath); A sequence that would cause ctime to change as a result of nlink changes. A sequence that might be appropriate in some cases even inside the filesystem code.
If you read my first posting on this thread you will see that i do see some, albeit questionable, value to the ctime update as a means of spotting the fact something relating to the inode has changed. While not technically required it is not necessarily wrong to update ctime. SUSv3 even allows for it and for some filesystems it would positively be the right thing to do.
PS. The name is J.W. or JW and has never been John.
Quoted 100 lines
Hans
Quoted 83 lines · Alex Adriaanse
Hans,
Yes, I agree with J.W. However, I also think that Andrew has a good point in that the behavior across Linux filesystems (ReiserFS, ext2, ext3, minix, etc.) should be consistent. Either they should all update ctime during renames, or none of them should.
Anyway, I'll try to work with the GNU tar maintainer to get this problem in tar fixed. It'll probably be a lot harder to fix in tar than to have ReiserFS update ctimes since it'll require major changes in
the --listed-incremental snapshot files. However, if you don't think it's a
good idea to make these changes to ReiserFS then we'll just work on fixing up tar.Thanks,
Alex
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Hans Reiser
Sent: Monday, October 13, 2003 3:46 AM
To: Alex Adriaanse
Cc: jw schultz; Linux Kernel Mailing List; [email protected]
Subject: Re: ReiserFS patch for updating ctimes of renamed filesAlex, are you convinced by jw? (I think I am.) Would you be willing to submit a patch for tar instead?
Hans
Quoted 14 lines · jw schultz
On Mon, Oct 13, 2003 at 09:49:20AM +0400, Hans Reiser wrote:
In theory it is cleaner and purer to do it the way we did. In practice,
Quoted 7 lines
Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
Quoted 16 lines
Updating ctime does seem messy and a bit irrelevant for the atomic rename. You are modifying the directories not the fricken file. This isn't DOS! But it would seem he does indeed have an issue although i'm not sure what. I've never used the listed-incremental option of tar and since the manpage is incomplete <rant deleted> i don't know what it actually does. However, i have found the use of ctime to be terribly unreliable for file management and given what the standards have to say on the issue it sounds like tar is being abused or has a bug.
--
Hans-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/Signature · 8 lines
Hans - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 7 lines · jw schultz
Of course if i had designed it in the first place with the filesystem semantics that we have now there might be no rename system call. Renames would be done by
link(oldpath, newpath); unlink(oldpath); A sequence that would cause ctime to change as a result of nlink changes. A sequence that might be appropriate in some cases even inside the filesystem code.
Once upon a time, that's how renames were always done.
The rename() system call was added because (a) it provides the additional atomicity semantic, which link+unlink does not; (b) it is useful to allow directory renames, but directory hard links are dangerous so not allowed any more.
-- Jamie
Quoted 5 lines · Hans Reiser
In theory it is cleaner and purer to do it the way we did. In practice,
Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
It would be best to make this change. minix, ext2 and ext3 do set ctime, so it is "the Linux standard".
btw, this code:
old_dir->i_ctime = old_dir->i_mtime = CURRENT_TIME;
new_dir->i_ctime = new_dir->i_mtime = CURRENT_TIME;
old_inode->i_ctime = CURRENT_TIME;should avoid evaluating CURRENT_TIME three times: is has some computational cost and if an interrupt happens at the wrong time you end up with differing times in the inode(s).
Quoted 15 lines · Andrew Morton
Quoted 3 lines · Hans Reiser
In theory it is cleaner and purer to do it the way we did. In practice, Alex's problem seems like a real one, and I don't know how hard it is to change tar to do the right thing. We'll discuss it in a small seminar today.
It would be best to make this change. minix, ext2 and ext3 do set ctime, so it is "the Linux standard".
do you think schultz's arguments about why it is wrong are correct? They seem well thought out to me.
Signature · 1 line
Hans
Quoted 3 lines · Hans Reiser
do you think schultz's arguments about why it is wrong are correct? They seem well thought out to me.
Well given that you've renamed the file, you do want the backup program to pick up the "new" file. But it'd be a pretty lame backup program which was fooled by a missing ctime update.
Yes, John has a point but we're not going to go and change all the other filesystems (are we?).
Quoted 7 lines · Andrew Morton
Quoted 3 lines · Hans Reiser
do you think schultz's arguments about why it is wrong are correct? They seem well thought out to me.
Quoted 11 lines
Well given that you've renamed the file, you do want the backup program to pick up the "new" file. But it'd be a pretty lame backup program which was fooled by a missing ctime update.
Yes, John has a point but we're not going to go and change all the other filesystems (are we?).
why not? It is correct therefor....
Signature · 1 line
Hans
Quoted 9 lines · Hans Reiser
Quoted 8 lines
Yes, John has a point but we're not going to go and change all the other filesystems (are we?).
why not? It is correct therefor....
Because that's the way we've always done it and there have been zero complaints about it and changing it risks breaking things.
I should think that's pretty obvious?
Quoted 11 lines · Andrew Morton
Quoted 3 lines · Hans Reiser
do you think schultz's arguments about why it is wrong are correct? They seem well thought out to me.
Well given that you've renamed the file, you do want the backup program to pick up the "new" file. But it'd be a pretty lame backup program which was fooled by a missing ctime update.
Yes, John has a point but we're not going to go and change all the other filesystems (are we?).
I don't know who John is but i sure hope we are not going to go changing how working filesystems function. It may be technically correct to not update ctime but that doesn't mean that it is incorrect to update it either. It all depends on the filesystem. They aren't all the same. We have some that don't support symlinks or hardlinks or that have or lack other features.
<OT>
There are change detections through timestamp (mtime) i am concerned about. As an rsync maintainer i worry about be extended attributes or ACLs changing with no modifiable timestamps being updated. And, by the way, because you cannot set ctime it doesn't qualify. Then there is jfs which i found did not update mtime when directories change unless the block count changes, but that might have been fixed already.
</OT>
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Vladimir will look at this and get back to you. Thanks kindly for this.
Hans
Quoted 50 lines · Alex Adriaanse
Hi,
I ran into some trouble trying to do incremental backups with GNU tar (using --listed-incremental) where renaming a file in between backups would cause the file to disappear upon restoration. When investigating the issue I discovered that this doesn't happen on ext2, ext3, and tmpfs filesystems but only on ReiserFS filesystems. I also noticed that for example ext3 updates the affected file's ctime upon rename whereas ReiserFS doesn't, so I'm thinking this causes tar to believe that the file existed before the first backup was taking under the new name, and as a result it doesn't back it up during the second backup. So I believe ReiserFS needs to update ctimes for renamed files in order for incremental GNU tar backups to work reliably.
I made some changes to the
reiserfs_renamefunction that I *think* should fix the problem. However, I don't know much about ReiserFS's internals, and I haven't been able to test them out to see if things work now since I can't afford to deal with potential FS corruption with my current Linux box.I included a patch below against the 2.4.22 kernel with my changes. Would somebody mind taking a look at this to see if I did things right here (and perhaps wouldn't mind testing it out either)? If it works then I (and I'm sure others who've experienced the same problem) would like to see the changes applied to the next 2.4.x (and 2.6.x?) release.
Thanks a lot.
Alex
reiserfs/namei.c Sun Oct 12 00:39:05 2003 renamed from reiserfs/namei.c.orig Mon Aug 25 06:44:43 2003 +2
-1@@ -1207,6 +1207,8 @@ journal_mark_dirty (&th, old_dir->i_sb, old_de.de_bh); old_dir->i_ctime = old_dir->i_mtime = CURRENT_TIME; new_dir->i_ctime = new_dir->i_mtime = CURRENT_TIME; + old_inode->i_ctime = CURRENT_TIME; + reiserfs_update_sd (&th, old_inode); if (new_dentry_inode) { // adjust link number of the victim -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
Hans