Skip to contentSkip to conversation

i_version changes

DISCUSSION Christoph Hellwig 7 participants 15 messages Feb 10, 2008 – Feb 15, 2008 VFS

lore
Share & tools
lore
#0 lore raw

I think the i_version changes that hit mainline about a week ago are not as nice as they should be.

First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for i_version on regular files are supposed to be, and how it differes from the existing semantics for directories.

Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert file_update_time to update c and mtime through ->setattr anyway.

Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.

Beyond those any good reason for making inode_inc_iversion inline, especially after the first patch introduced it properly out of line.

And as a last note please stop pushing these kind of core changes through specific filesystem trees. If this had been in ->mm we would have caught this a lot earlier, and would have also meant you'd get input and possible even implementations from other filesystem maintainers.

#1 lore raw
Quoted 8 lines · Christoph Hellwig

I think the i_version changes that hit mainline about a week ago are not as nice as they should be.

First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for i_version on regular files are supposed to be, and how it differes from the existing semantics for directories.

Quoted 4 lines

Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert file_update_time to update c and mtime through ->setattr anyway.

I don't understand that comment. (What is "this" in the second sentence, for example?)

Quoted 9 lines

Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.

That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?

The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or i_version as the change attribute.

--b.

Quoted 12 lines

Beyond those any good reason for making inode_inc_iversion inline, especially after the first patch introduced it properly out of line.

And as a last note please stop pushing these kind of core changes through specific filesystem trees. If this had been in ->mm we would have caught this a lot earlier, and would have also meant you'd get input and possible even implementations from other filesystem maintainers.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 10 lines · J. Bruce Fields
Quoted 8 lines · Christoph Hellwig

Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.

Quoted 2 lines

That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?

One of the reasons NOT to make it mandatory is that it forces updates of the inode after every write. On ext3/ext4 this is expensive, as the ext3_dirty_inode() packs the inode from memory into the buffer each time, so that it can be journaled.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#3 lore raw
In reply to J. Bruce Fields
Quoted 5 lines · J. Bruce Fields
Quoted 5 lines

First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for i_version on regular files are supposed to be, and how it differes from the existing semantics for directories.

Quoted 7 lines
Quoted 4 lines

Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert file_update_time to update c and mtime through ->setattr anyway.

I don't understand that comment. (What is "this" in the second sentence, for example?)

i_version. Instead of hardcoding i_version updates in file_update_time it would be better to add an ia_verstion to struct iattr and update it that way.

Quoted 9 lines
Quoted 8 lines

Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.

Quoted 6 lines

That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?

The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or i_version as the change attribute.

Probably through export_operations somehow. Andreas mentioned in the other reply that he wants it only conditionally due to the overhead on extN, and enabling this from an export operation called when nfs exporting a filesystem.

Btw, stupid question: the commit message for the i_version changes mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.

#4 lore raw
Quoted 6 lines · Christoph Hellwig

Btw, stupid question: the commit message for the i_version changes mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.

Support for 64-bit on-disk time stamps alone does not suffice. In order to label all file/directory changes unambiguously, you would also need nanosecond timekeeping support.

An example is XFS, which has had on-disk support for 64-bit time stamps since forever, but is in practice limited by the Linux default 250Hz internal clock. We've seen plenty of examples of NFS clients missing updates on the resulting filesystem due to the fact that they occurred within 1/250 sec of each other.

Cheers
Trond

#5 lore raw
Quoted 12 lines · Trond Myklebust
Quoted 6 lines · Christoph Hellwig

Btw, stupid question: the commit message for the i_version changes mentions this is to work around lack of granularity for ctime updates. But all modern filesystems (and I includ ext4 in that here) have 64bit timestamps already, so that should be enough. It would certainly avoid all this additional code, and especially the additional space used in struct inode which can hurt quite a lot.

Support for 64-bit on-disk time stamps alone does not suffice. In order to label all file/directory changes unambiguously, you would also need nanosecond timekeeping support.

Quoted 5 lines

An example is XFS, which has had on-disk support for 64-bit time stamps since forever, but is in practice limited by the Linux default 250Hz internal clock. We've seen plenty of examples of NFS clients missing updates on the resulting filesystem due to the fact that they occurred within 1/250 sec of each other.

The other issue which unfortunately makes ctime a non-starter is the ability of ctime to go backward due to clock changes.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#6 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines

First there's a complete lack of documentation on this, which is very bad. Please document what the new semantics for i_version on regular files are supposed to be, and how it differes from the existing semantics for directories.

Second abusing one of the rather scare superblock mount flags is a bad idea. It would be much better to set this through ->setattr and an extension of struct iattr. Especially as we need to convert file_update_time to update c and mtime through ->setattr anyway.

Quoted 6 lines
Quoted 2 lines

I don't understand that comment. (What is "this" in the second sentence, for example?)

i_version. Instead of hardcoding i_version updates in file_update_time it would be better to add an ia_verstion to struct iattr and update it that way.

Got it, thanks.

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines

Third using the MS_ flag but then actually having a filesystem mount option to enable it is more than confusing. After all MS_ options (at least the exported parts) are the mount ABI for common options. Also this option doesn't show up in ->show_options, which is something Miklos will beat you up for :)
I'm also not convinced this should be option behaviour, either you do update i_version for a given filesystem or you don't - having an obscure mount option will only give you confusion.

That does sound likely to be confusing. Any chance we could just make the new behavior mandatory?

Quoted 8 lines
Quoted 3 lines

The one thing we need in nfsd is just an easy (in-kernel) way to check whether a given filesystem supports this, so nfsd can decide whether to use ctime or i_version as the change attribute.

Probably through export_operations somehow. Andreas mentioned in the other reply that he wants it only conditionally due to the overhead on extN, and enabling this from an export operation called when nfs exporting a filesystem.

It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.

So I'm inclined to think this should be an option that's permanently associated with the filesystem, rather than something that's changed on or off by nfsd (or even on or off with each mount).

--b.

#7 lore raw
Quoted 3 lines · J. Bruce Fields

It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.

For NFSD's needs, it is only necessary that changes in i_version that are potentially visible over NFS actually be stored on disk.

You could come up with an interface where NFSD sets a flag when it reads i_version, and changes to the file only change i_version if the flag is set (at which point the flag is cleared).

This would give fully correct NFS semantics, and no overhead when NFS access is not in use

This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of i_version.

Of course then the semantics of the new i_version are very different to the old i_version, so maybe we need two fields in the inode....

NeilBrown

#8 lore raw
Quoted 11 lines · NeilBrown
Quoted 4 lines · J. Bruce Fields

It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.

For NFSD's needs, it is only necessary that changes in i_version that are potentially visible over NFS actually be stored on disk.

Quoted 8 lines

You could come up with an interface where NFSD sets a flag when it reads i_version, and changes to the file only change i_version if the flag is set (at which point the flag is cleared).

This would give fully correct NFS semantics, and no overhead when NFS access is not in use

I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of i_version which is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.

Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?

Thanx...

ps

Quoted 13 lines

This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of i_version.

Of course then the semantics of the new i_version are very different to the old i_version, so maybe we need two fields in the inode....

NeilBrown

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 6 lines · Peter Staubach

I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of i_version which is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.

As I said, the "NFS has seen this i_version" flag needs to be on stable storage, e.g. the lsb of the i_version. This will ensure that any change after NFSD saw the i_version will cause the i_version to be updated.
So I think it can provide correct semantics.
Precise details:
NFSD: when reading i_version

      take lock
      tmp = i_version
      i_version |= 1
      drop lock
      return tmp & ~1;
  VFS when making any change:
      take lock
      if (i_version & 1) {
           i_version++;
           changed=1
      }
      drop lock
      if changed, sync inode
Quoted 6 lines

Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?

Correct NFS semantics require that the i_version be written to disk before (or when) the change is committed. That means lots more inodes in the journal.
If you are already doing data=journal, it the hit probably isn't too high.(?)

You are right: measuring the cost is important. However as we are designing a generic filesystem interface, we need to understand the cost on multiple filesystems in a variety of configuration .... or give the filesystem complete information and let it decide the optimal implementation.

Giving the filesystem full information means having an inode_operation "nfsd_reads_version" which returns the number to be used as change_id.

NeilBrown

#10 lore raw
Quoted 22 lines · NeilBrown
Quoted 6 lines · Peter Staubach

I don't think that this is quite true. If the file is changed when the NFS server is not running, then the value of i_version which is used when the NFS server starts up again must be different than the value which was previously used when the NFS server was previously running.

As I said, the "NFS has seen this i_version" flag needs to be on stable storage, e.g. the lsb of the i_version. This will ensure that any change after NFSD saw the i_version will cause the i_version to be updated.
So I think it can provide correct semantics.
Precise details:
NFSD: when reading i_version

      take lock
      tmp = i_version
      i_version |= 1
      drop lock
      return tmp & ~1;
Quoted 10 lines
  VFS when making any change:
      take lock
      if (i_version & 1) {
           i_version++;
           changed=1
      }
      drop lock
      if changed, sync inode

Yes, this does seem like it would do the job. It could perhaps be optimized somewhat to avoid lock contention, but I do think that this would suffice.

Quoted 8 lines
Quoted 7 lines

Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?

Quoted 7 lines

Correct NFS semantics require that the i_version be written to disk before (or when) the change is committed. That means lots more inodes in the journal.
If you are already doing data=journal, it the hit probably isn't too high.(?)

Correct NFS semantics also require that any modified metadata, including file times and file size, also be written to stable storage. Isn't this just another piece of modified metadata that would go hand-in-hand with updated file times?

We should also require that the file mtime change when the contents of the file are modified. This should happen whether or not the clock has ticked. Unfortunately, to implement this, we would need file time resolutions which are smaller granularity than the system clock. We could probably get away with nano- second resolutions in the file system.

Thanx...

ps

Quoted 13 lines

You are right: measuring the cost is important. However as we are designing a generic filesystem interface, we need to understand the cost on multiple filesystems in a variety of configuration .... or give the filesystem complete information and let it decide the optimal implementation.

Giving the filesystem full information means having an inode_operation "nfsd_reads_version" which returns the number to be used as change_id.

NeilBrown

#11 lore raw
In reply to Peter Staubach

hi,

Quoted 6 lines · Peter Staubach

Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?

Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the i_version flag.
http://bullopensource.org/ext4/20071116/ffsb-write.html

regards.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
Quoted 4 lines · Jean noel Cordenner

hi,

Quoted 1 line · Peter Staubach
Quoted 8 lines
Quoted 5 lines

Is the perceived performance hit really going to be as large as suspected? We already update the time fields fairly often and we don't pay a huge penalty for those, or at least not a penalty that we aren't willing to pay. Has anyone measured the cost?

Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the i_version flag.
http://bullopensource.org/ext4/20071116/ffsb-write.html

This is good information.

A couple of questions -- what is the "-I 256" option used for the ext4 mkfs?

What was the variance between the results of the 5 runs? Is 2% significant or not?

Thanx...

ps
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw
Quoted 8 lines · Peter Staubach
Quoted 2 lines

Few month ago, I ran a FFSB test on a 2.6.23 kernel enabling or not the i_version flag.
http://bullopensource.org/ext4/20071116/ffsb-write.html

This is good information.

A couple of questions -- what is the "-I 256" option used for the ext4 mkfs?

This option force the inode size to 256Bytes instead of 128. The requirement is to have a 64bits i_version counter, therefore the ext4 inode as to be extended to 256Bytes.

Quoted 3 lines

What was the variance between the results of the 5 runs? Is 2% significant or not?

I didn't keep the raw results, but they were all very close to the average (less than 1% of the average).

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to NeilBrown
Quoted 13 lines · NeilBrown
Quoted 3 lines · J. Bruce Fields

It's not OK to update it only sometimes. If updates are made while nfsd isn't running, those needed to be reflected in the change attribute, so the changes aren't missed when nfsd comes back up.

For NFSD's needs, it is only necessary that changes in i_version that are potentially visible over NFS actually be stored on disk.

You could come up with an interface where NFSD sets a flag when it reads i_version, and changes to the file only change i_version if the flag is set (at which point the flag is cleared).

Quoted 8 lines

This would give fully correct NFS semantics, and no overhead when NFS access is not in use

This flag would need to be stored in stable storage too, so probably easiest to make it the least significant bit of i_version.

Of course then the semantics of the new i_version are very different to the old i_version, so maybe we need two fields in the inode....

Sure. I'd rather have a per-superblock "use i_version" bit than a per-inode "use i_version" bit, just for simplicity's sake, unless there's a really convincing case that it helps.

--b.