i_version changes
I think the i_version changes that hit mainline about a week ago are not as nice as they should be.
First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for i_version on regular files are supposed to be, and how it differes from the existing semantics for directories.
Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert file_update_time to update c and mtime through ->setattr anyway.
Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.
Beyond those any good reason for making inode_inc_iversion inline, especially after the first patch introduced it properly out of line.
And as a last note please stop pushing these kind of core changes through specific filesystem trees. If this had been in ->mm we would have caught this a lot earlier, and would have also meant you'd get input and possible even implementations from other filesystem maintainers.
Quoted 8 lines · Christoph Hellwig
I think the
i_versionchanges that hit mainline about a week ago are not as nice as they should be.First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for
i_versionon regular files are supposed to be, and how it differes from the existing semantics for directories.
Quoted 4 lines
Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert
file_update_timeto update c and mtime through ->setattr anyway.
I don't understand that comment. (What is "this" in the second sentence, for example?)
Quoted 9 lines
Third using the
MS_flag but then actually having a filesystem mount option to enable it is more than confusing. After allMS_options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do updatei_versionfor a given filesystem or you don't - having an obscure mount option will only give you confusion.
That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?
The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or i_version as the change attribute.
--b.
Quoted 12 lines
Beyond those any good reason for making
inode_inc_iversioninline, especially after the first patch introduced it properly out of line.And as a last note please stop pushing these kind of core changes through specific filesystem trees. If this had been in ->mm we would have caught this a lot earlier, and would have also meant you'd get input and possible even implementations from other filesystem maintainers.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · J. Bruce Fields
Quoted 8 lines · Christoph Hellwig
Third using the
MS_flag but then actually having a filesystem mount option to enable it is more than confusing. After allMS_options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do updatei_versionfor a given filesystem or you don't - having an obscure mount option will only give you confusion.
Quoted 2 lines
That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?
One of the reasons NOT to make it mandatory is that it forces updates of the inode after every write. On ext3/ext4 this is expensive, as the ext3_dirty_inode() packs the inode from memory into the buffer each time, so that it can be journaled.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 5 lines · J. Bruce Fields
Quoted 5 lines
First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for
i_versionon regular files are supposed to be, and how it differes from the existing semantics for directories.
Quoted 7 lines
Quoted 4 lines
Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert
file_update_timeto update c and mtime through ->setattr anyway.I don't understand that comment. (What is "this" in the second sentence, for example?)
i_version. Instead of hardcoding i_version updates in file_update_time it would be better to add an ia_verstion to struct iattr and update it that way.
Quoted 9 lines
Quoted 8 lines
Third using the
MS_flag but then actually having a filesystem mount option to enable it is more than confusing. After allMS_options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do updatei_versionfor a given filesystem or you don't - having an obscure mount option will only give you confusion.
Quoted 6 lines
That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?
The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or
i_versionas the change attribute.
Probably through export_operations somehow. Andreas mentioned in the other reply that he wants it only conditionally due to the overhead on extN, and enabling this from an export operation called when nfs exporting a filesystem.
Btw, stupid question: the commit message for the i_version changes mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.
Quoted 6 lines · Christoph Hellwig
Btw, stupid question: the commit message for the
i_versionchanges mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.
Support for 64-bit on-disk time stamps alone does not suffice. In order to label all file/directory changes unambiguously, you would also need nanosecond timekeeping support.
An example is XFS, which has had on-disk support for 64-bit time stamps since forever, but is in practice limited by the Linux default 250Hz internal clock. We've seen plenty of examples of NFS clients missing updates on the resulting filesystem due to the fact that they occurred within 1/250 sec of each other.
Cheers
Trond
Quoted 12 lines · Trond Myklebust
Quoted 6 lines · Christoph Hellwig
Btw, stupid question: the commit message for the
i_versionchanges mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.Support for 64-bit on-disk time stamps alone does not suffice. In order to label all file/directory changes unambiguously, you would also need nanosecond timekeeping support.
Quoted 5 lines
An example is XFS, which has had on-disk support for 64-bit time stamps since forever, but is in practice limited by the Linux default 250Hz internal clock. We've seen plenty of examples of NFS clients missing updates on the resulting filesystem due to the fact that they occurred within 1/250 sec of each other.
The other issue which unfortunately makes ctime a non-starter is the ability of ctime to go backward due to clock changes.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines
First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for
i_versionon regular files are supposed to be, and how it differes from the existing semantics for directories.Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert
file_update_timeto update c and mtime through ->setattr anyway.
Quoted 6 lines
Quoted 2 lines
I don't understand that comment. (What is "this" in the second sentence, for example?)
i_version. Instead of hardcodingi_versionupdates infile_update_timeit would be better to add ania_verstionto struct iattr and update it that way.
Got it, thanks.
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Third using the
MS_flag but then actually having a filesystem mount option to enable it is more than confusing. After allMS_options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do updatei_versionfor a given filesystem or you don't - having an obscure mount option will only give you confusion.That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?
Quoted 8 lines
Quoted 3 lines
The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or
i_versionas the change attribute.Probably through
export_operationssomehow. Andreas mentioned in the other reply that he wants it only conditionally due to the overhead on extN, and enabling this from an export operation called when nfs exporting a filesystem.
It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.
So I'm inclined to think this should be an option that's permanently associated with the filesystem, rather than something that's changed on or off by nfsd (or even on or off with each mount).
--b.
Quoted 3 lines · J. Bruce Fields
It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.
For NFSD's needs, it is only necessary that changes in i_version that are potentially visible over NFS actually be stored on disk.
You could come up with an interface where NFSD sets a flag when it reads i_version, and changes to the file only change i_version if the flag is set (at which point the flag is cleared).
This would give fully correct NFS semantics, and no overhead when NFS access is not in use
This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of i_version.
Of course then the semantics of the new i_version are very different to the old i_version, so maybe we need two fields in the inode....
NeilBrown
Quoted 11 lines · NeilBrown
Quoted 4 lines · J. Bruce Fields
It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.
For NFSD's needs, it is only necessary that changes in
i_versionthat are potentially visible over NFS actually be stored on disk.
Quoted 8 lines
You could come up with an interface where NFSD sets a flag when it reads
i_version, and changes to the file only changei_versionif the flag is set (at which point the flag is cleared).This would give fully correct NFS semantics, and no overhead when NFS access is not in use
I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of i_version which is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.
Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?
Thanx...
ps
Quoted 13 lines
This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of
i_version.Of course then the semantics of the new
i_versionare very different to the oldi_version, so maybe we need two fields in the inode....NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Peter Staubach
I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of
i_versionwhich is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.
As I said, the "NFS has seen this i_version" flag needs to be on stable storage, e.g. the lsb of the i_version. This will ensure that any change after NFSD saw the i_version will cause the i_version to be updated.
So I think it can provide correct semantics.
Precise details:
NFSD: when reading i_version
take lock
tmp = i_version
i_version |= 1
drop lock
return tmp & ~1; VFS when making any change:
take lock
if (i_version & 1) {
i_version++;
changed=1
}
drop lock
if changed, sync inodeQuoted 6 lines
Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?
Correct NFS semantics require that the i_version be written to disk before (or when) the change is committed. That means lots more inodes in the journal.
If you are already doing data=journal, it the hit probably isn't too high.(?)
You are right: measuring the cost is important. However as we are designing a generic filesystem interface, we need to understand the cost on multiple filesystems in a variety of configuration .... or give the filesystem complete information and let it decide the optimal implementation.
Giving the filesystem full information means having an inode_operation "nfsd_reads_version" which returns the number to be used as change_id.
NeilBrown
Quoted 22 lines · NeilBrown
Quoted 6 lines · Peter Staubach
I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of
i_versionwhich is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.As I said, the "NFS has seen this
i_version" flag needs to be on stable storage, e.g. the lsb of thei_version. This will ensure that any change after NFSD saw thei_versionwill cause thei_versionto be updated.
So I think it can provide correct semantics.
Precise details:
NFSD: when readingi_versiontake lock tmp = i_version i_version |= 1 drop lock return tmp & ~1;
Quoted 10 lines
VFS when making any change:
take lock
if (i_version & 1) {
i_version++;
changed=1
}
drop lock
if changed, sync inodeYes, this does seem like it would do the job. It could perhaps be optimized somewhat to avoid lock contention, but I do think that this would suffice.
Quoted 8 lines
Quoted 7 lines
Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?
Quoted 7 lines
Correct NFS semantics require that the
i_versionbe written to disk before (or when) the change is committed. That means lots more inodes in the journal.
If you are already doing data=journal, it the hit probably isn't too high.(?)
Correct NFS semantics also require that any modified metadata, including file times and file size, also be written to stable storage. Isn't this just another piece of modified metadata that would go hand-in-hand with updated file times?
We should also require that the file mtime change when the contents of the file are modified. This should happen whether or not the clock has ticked. Unfortunately, to implement this, we would need file time resolutions which are smaller granularity than the system clock. We could probably get away with nano- second resolutions in the file system.
Thanx...
ps
Quoted 13 lines
You are right: measuring the cost is important. However as we are designing a generic filesystem interface, we need to understand the cost on multiple filesystems in a variety of configuration .... or give the filesystem complete information and let it decide the optimal implementation.
Giving the filesystem full information means having an
inode_operation"nfsd_reads_version" which returns the number to be used aschange_id.NeilBrown
hi,
Quoted 6 lines · Peter Staubach
Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?
Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the i_version flag.
http://bullopensource.org/ext4/20071116/ffsb-write.html
regards.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Jean noel Cordenner
hi,
Quoted 1 line · Peter Staubach
Quoted 8 lines
Quoted 5 lines
Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?
Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the
i_versionflag.
http://bullopensource.org/ext4/20071116/ffsb-write.html
This is good information.
A couple of questions -- what is the "-I 256" option used for the ext4 mkfs?
What was the variance between the results of the 5 runs? Is 2% significant or not?
Thanx...
ps
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Peter Staubach
Quoted 2 lines
Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the
i_versionflag.
http://bullopensource.org/ext4/20071116/ffsb-write.htmlThis is good information.
A couple of questions -- what is the "-I 256" option used for the ext4 mkfs?
This option force the inode size to 256Bytes instead of 128. The requirement is to have a 64bits i_version counter, therefore the ext4 inode as to be extended to 256Bytes.
Quoted 3 lines
What was the variance between the results of the 5 runs? Is 2% significant or not?
I didn't keep the raw results, but they were all very close to the average (less than 1% of the average).
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 13 lines · NeilBrown
Quoted 3 lines · J. Bruce Fields
It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.
For NFSD's needs, it is only necessary that changes in
i_versionthat are potentially visible over NFS actually be stored on disk.You could come up with an interface where NFSD sets a flag when it reads
i_version, and changes to the file only changei_versionif the flag is set (at which point the flag is cleared).
Quoted 8 lines
This would give fully correct NFS semantics, and no overhead when NFS access is not in use
This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of
i_version.Of course then the semantics of the new
i_versionare very different to the oldi_version, so maybe we need two fields in the inode....
Sure. I'd rather have a per-superblock "use i_version" bit than a per-inode "use i_version" bit, just for simplicity's sake, unless there's a really convincing case that it helps.
--b.