make "noac" and "actimeo=0" work correctly

Hi.

I've been looking at a bugzilla which describes a problem where a customer was advised to use either the "noac" or "actimeo=0" mount options to solve a consistency problem that they were seeing in the file attributes. It turned out that this solution did not work reliably for them because sometimes, the local attribute cache was believed to be valid and not timed out. (With an attribute cache timeout of 0, the cache should always appear to be timed out.)

In looking at this situation, it appears to me that the problem is that the attribute cache timeout code has an off-by-one error in it. It is assuming that the cache is valid in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo]. The cache should be considered valid only in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo). With this change, the options, "noac" and "actimeo=0", work as originally expected.

While I was there, I addressed a problem with the jiffies overflowing on 32 bit systems. When overflow occurs, the value of read_cache_jiffies + attrtimeo can be less then the value of read_cache_jiffies. This would cause an unnecessary GETATTR over the wire.

Thoughts and/or comments? This is an updated patch which includes the previous support which was added to correct the noac/actimeo=0 handling.

Signed-off-by Peter Staubach <[email protected]>

fs/nfs/dir.c renamed from fs/nfs/dir.c.org +2 -1

@@ -1808,7 +1808,7 @@ static int nfs_access_get_cached(struct
       cache = nfs_access_search_rbtree(inode, cred);
       if (cache == NULL)
               goto out;
-       if (!time_in_range(jiffies, cache->jiffies, cache->jiffies +
 nfsi->attrtimeo))
+       if (!nfs_time_in_range_open(jiffies, cache->jiffies, cache->jiffies
+ nfsi->attrtimeo))
               goto out_stale;
       res->jiffies = cache->jiffies;
       res->cred = cache->cred;

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -8

@@ -706,14 +706,7 @@ int nfs_attribute_timeout(struct inode *
 
       if (nfs_have_delegation(inode, FMODE_READ))
               return 0;
-       /*
-        * Special case: if the attribute timeout is set to 0, then always
-        *               treat the cache as having expired (unless holding
-        *               a delegation).
-        */
-       if (nfsi->attrtimeo == 0)
-               return 1;
-       return !time_in_range(jiffies, nfsi->read_cache_jiffies,
 nfsi->read_cache_jiffies + nfsi->attrtimeo);
+       return !nfs_time_in_range_open(jiffies, nfsi->read_cache_jiffies,
 nfsi->read_cache_jiffies + nfsi->attrtimeo);
 }

/**

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -1

@@ -1098,7 +1091,7 @@ static int nfs_update_inode(struct inode
               nfsi->attrtimeo_timestamp = now;
               nfsi->last_updated = now;
       } else {
-               if (!time_in_range(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
+               if (!nfs_time_in_range_open(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
                       if ((nfsi->attrtimeo <<= 1) >

NFS_MAXATTRTIMEO(inode))

                               nfsi->attrtimeo = NFS_MAXATTRTIMEO(inode);
                       nfsi->attrtimeo_timestamp = now;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +1 -1

@@ -121,7 +121,7 @@ struct nfs_inode {
        *
        * We need to revalidate the cached attrs for this inode if
        *
-        *      jiffies - read_cache_jiffies > attrtimeo
+        *      jiffies - read_cache_jiffies >= attrtimeo
        */
       unsigned long           read_cache_jiffies;
       unsigned long           attrtimeo;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +8 -0

+        *
+        * Otherwise, just check to see whether a is in [b, c).
+        */
+       if (c < b)
+               return time_after_eq(a, b) || time_before(a, c);
+       return time_after_eq(a, b) && time_before(a, c);
+}
+
 static inline int NFS_STALE(const struct inode *inode)
 {
       return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);

Hi Peter-

Quoted 27 lines

Hi.

I've been looking at a bugzilla which describes a problem where a customer was advised to use either the "noac" or "actimeo=0" mount options to solve a consistency problem that they were seeing in the file attributes. It turned out that this solution did not work reliably for them because sometimes, the local attribute cache was believed to be valid and not timed out. (With an attribute cache timeout of 0, the cache should always appear to be timed out.)

In looking at this situation, it appears to me that the problem is that the attribute cache timeout code has an off-by-one error in it. It is assuming that the cache is valid in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo]. The cache should be considered valid only in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo). With this change, the options, "noac" and "actimeo=0", work as originally expected.

While I was there, I addressed a problem with the jiffies overflowing on 32 bit systems. When overflow occurs, the value of read_cache_jiffies + attrtimeo can be less then the value of read_cache_jiffies. This would cause an unnecessary GETATTR over the wire.

Quoted 4 lines

Thoughts and/or comments? This is an updated patch which includes the previous support which was added to correct the noac/actimeo=0 handling.

A couple of random thoughts below.

Quoted 63 lines

Signed-off-by Peter Staubach <[email protected]>

fs/nfs/dir.c renamed from fs/nfs/dir.c.org +2 -1

@@ -1808,7 +1808,7 @@ static int nfs_access_get_cached(struct
       cache = nfs_access_search_rbtree(inode, cred);
       if (cache == NULL)
               goto out;
-       if (!time_in_range(jiffies, cache->jiffies, cache->jiffies +
 nfsi->attrtimeo))
+       if (!nfs_time_in_range_open(jiffies, cache->jiffies, cache->jiffies
+ nfsi->attrtimeo))
               goto out_stale;
       res->jiffies = cache->jiffies;
       res->cred = cache->cred;

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -8

@@ -706,14 +706,7 @@ int nfs_attribute_timeout(struct inode *
 
       if (nfs_have_delegation(inode, FMODE_READ))
               return 0;
-       /*
-        * Special case: if the attribute timeout is set to 0, then always
-        *               treat the cache as having expired (unless holding
-        *               a delegation).
-        */
-       if (nfsi->attrtimeo == 0)
-               return 1;
-       return !time_in_range(jiffies, nfsi->read_cache_jiffies,
 nfsi->read_cache_jiffies + nfsi->attrtimeo);
+       return !nfs_time_in_range_open(jiffies, nfsi->read_cache_jiffies,
 nfsi->read_cache_jiffies + nfsi->attrtimeo);
 }

/**

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -1

@@ -1098,7 +1091,7 @@ static int nfs_update_inode(struct inode
               nfsi->attrtimeo_timestamp = now;
               nfsi->last_updated = now;
       } else {
-               if (!time_in_range(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
+               if (!nfs_time_in_range_open(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
                       if ((nfsi->attrtimeo <<= 1) >

NFS_MAXATTRTIMEO(inode))

                               nfsi->attrtimeo = NFS_MAXATTRTIMEO(inode);
                       nfsi->attrtimeo_timestamp = now;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +1 -1

@@ -121,7 +121,7 @@ struct nfs_inode {
        *
        * We need to revalidate the cached attrs for this inode if
        *
-        *      jiffies - read_cache_jiffies > attrtimeo
+        *      jiffies - read_cache_jiffies >= attrtimeo
        */
       unsigned long           read_cache_jiffies;
       unsigned long           attrtimeo;

@@ -244,6 +244,22 @@ static inline unsigned NFS_MAXATTRTIMEO(
       return S_ISDIR(inode->i_mode) ? nfss->acdirmax : nfss->acregmax;
 }
 

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 3 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +2 -0

+static inline int nfs_time_in_range_open(unsigned long a,
+                               unsigned long b, unsigned long c)
 

All of nfs_time_in_range_open's callers use a sum of 'b' and 'nfsi->attrtimeo' for 'c'. Would it be cleaner to pass in nfsi->attrtimeo' rather than 'b + nfsi->attrtimeo' and do the sum here? It might make sense to explicitly check nfsi->attrtimeo for zero here to document the special behavior of "actimeo=0".

Alternately, checking explicitly if b and c are equal might accomplish the same without changing the synopsis.

Also, all of nfs_time_in_range_open's callers negate the return value.
Would the code in the callers be cleaner if that negation was moved
into nfs_time_in_range_open? You might rename nfs_time_in_range_open() as nfs_cache_has_expired(), for example, to make the 'if' statements in the callers make sense in English.

My feeling is that if you have to sit and stare at this for 5 minutes to understand precisely how it works, it has already become too obfuscated. In addition to fixing any bugs, I wonder if we can make it easier to understand and maintain as well.

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 7 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +6 -0

+{
+       /*
+        * If c is less then b, then the jiffies have wrapped.
+        * If so, then check to see if a is between b and the
+        * max jiffies value or between 0 and the value of c.
+        * This is the range between b and c.
 

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

You could then simplify this to "return b != c && time_in_range(a, b, c);" or something like that. Or if you negate the return value here:

static inline nfs_attributes_have_expired(unsigned long current,
unsigned long
start, unsigned long end)

{
        return (start == end) || !time_in_range(current, start, end);
}

My 0.02USD.

Quoted 12 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +8 -0

+        *
+        * Otherwise, just check to see whether a is in [b, c).
+        */
+       if (c < b)
+               return time_after_eq(a, b) || time_before(a, c);
+       return time_after_eq(a, b) && time_before(a, c);
+}
+
 static inline int NFS_STALE(const struct inode *inode)

 {
       return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);

Quoted 4 lines · Chuck Lever

Hi Peter-

Hi, Chuck.

Quoted 30 lines

Quoted 27 lines · Peter Staubach

Hi.

I've been looking at a bugzilla which describes a problem where a customer was advised to use either the "noac" or "actimeo=0" mount options to solve a consistency problem that they were seeing in the file attributes. It turned out that this solution did not work reliably for them because sometimes, the local attribute cache was believed to be valid and not timed out. (With an attribute cache timeout of 0, the cache should always appear to be timed out.)

In looking at this situation, it appears to me that the problem is that the attribute cache timeout code has an off-by-one error in it. It is assuming that the cache is valid in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo]. The cache should be considered valid only in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo). With this change, the options, "noac" and "actimeo=0", work as originally expected.

While I was there, I addressed a problem with the jiffies overflowing on 32 bit systems. When overflow occurs, the value of read_cache_jiffies + attrtimeo can be less then the value of read_cache_jiffies. This would cause an unnecessary GETATTR over the wire.

Quoted 8 lines

Quoted 4 lines

Thoughts and/or comments? This is an updated patch which includes the previous support which was added to correct the noac/actimeo=0 handling.

A couple of random thoughts below.

Some thoughts in response --

Quoted 67 lines

Quoted 67 lines

Signed-off-by Peter Staubach <[email protected]>

fs/nfs/dir.c renamed from fs/nfs/dir.c.org +1 -1

@@ -1808,7 +1808,7 @@ static int nfs_access_get_cached(struct
      cache = nfs_access_search_rbtree(inode, cred);
      if (cache == NULL)
              goto out;
-       if (!time_in_range(jiffies, cache->jiffies, cache->jiffies +
 nfsi->attrtimeo))
+       if (!nfs_time_in_range_open(jiffies, cache->jiffies,

cache->jiffies
+ nfsi->attrtimeo))

              goto out_stale;
      res->jiffies = cache->jiffies;
      res->cred = cache->cred;

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -2

@@ -706,14 +706,7 @@ int nfs_attribute_timeout(struct inode *
 
      if (nfs_have_delegation(inode, FMODE_READ))
              return 0;
-       /*
-        * Special case: if the attribute timeout is set to 0, then

always
- * treat the cache as having expired (unless
holding

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -5

-        *               a delegation).
-        */
-       if (nfsi->attrtimeo == 0)
-               return 1;
-       return !time_in_range(jiffies, nfsi->read_cache_jiffies,

nfsi->read_cache_jiffies + nfsi->attrtimeo);
+ return !nfs_time_in_range_open(jiffies, nfsi->read_cache_jiffies,
nfsi->read_cache_jiffies + nfsi->attrtimeo);
}

/**

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -1

@@ -1098,7 +1091,7 @@ static int nfs_update_inode(struct inode
              nfsi->attrtimeo_timestamp = now;
              nfsi->last_updated = now;
      } else {
-               if (!time_in_range(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
+               if (!nfs_time_in_range_open(now,
 nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {

                      if ((nfsi->attrtimeo <<= 1) >
NFS_MAXATTRTIMEO(inode))
                              nfsi->attrtimeo = NFS_MAXATTRTIMEO(inode);
                      nfsi->attrtimeo_timestamp = now;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +1 -1

@@ -121,7 +121,7 @@ struct nfs_inode {
       *
       * We need to revalidate the cached attrs for this inode if
       *
-        *      jiffies - read_cache_jiffies > attrtimeo
+        *      jiffies - read_cache_jiffies >= attrtimeo
       */
      unsigned long           read_cache_jiffies;
      unsigned long           attrtimeo;

@@ -244,6 +244,22 @@ static inline unsigned NFS_MAXATTRTIMEO(
      return S_ISDIR(inode->i_mode) ? nfss->acdirmax : nfss->acregmax;
 }
 

Quoted 11 lines

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 3 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +2 -0

+static inline int nfs_time_in_range_open(unsigned long a,
+                               unsigned long b, unsigned long c)
 

All of nfs_time_in_range_open's callers use a sum of 'b' and 'nfsi->attrtimeo' for 'c'. Would it be cleaner to pass in nfsi->attrtimeo' rather than 'b + nfsi->attrtimeo' and do the sum here? It might make sense to explicitly check nfsi->attrtimeo for zero here to document the special behavior of "actimeo=0".

The behavior of "actimeo=0" isn't any more special than "actimeo=1". It simply indicates that the attribute timeout is 0 jiffies long.

I thought about reducing the arguments, but it didn't seem to yield anything any clearer to me.

Quoted 24 lines

Alternately, checking explicitly if b and c are equal might accomplish the same without changing the synopsis.

Also, all of nfs_time_in_range_open's callers negate the return value.
Would the code in the callers be cleaner if that negation was moved
into nfs_time_in_range_open? You might rename nfs_time_in_range_open() as nfs_cache_has_expired(), for example, to make the 'if' statements in the callers make sense in English.

My feeling is that if you have to sit and stare at this for 5 minutes to understand precisely how it works, it has already become too obfuscated. In addition to fixing any bugs, I wonder if we can make it easier to understand and maintain as well.

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 8 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +6 -0

+{
+       /*
+        * If c is less then b, then the jiffies have wrapped.
+        * If so, then check to see if a is between b and the
+        * max jiffies value or between 0 and the value of c.
+        * This is the range between b and c.
 

Quoted 5 lines

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

Clearly, time_in_range() is not sufficient if the 'c' has wrapped. It only tests to see if a >=b and a <= c. If 'c' is less than 'b', then time_in_range() will return false.

I am reluctant to fix time_in_range() because I don't know that it is broken. It appears to me that it works for other uses, because otherwise, someone would have "fixed" it.

Quoted 13 lines

You could then simplify this to "return b != c && time_in_range(a, b, c);" or something like that. Or if you negate the return value here:

static inline nfs_attributes_have_expired(unsigned long current,
unsigned long
start, unsigned long end)

{
       return (start == end) || !time_in_range(current, start, end);
}

My 0.02USD.

The change, which makes attrtimeo=0 work for free, is to figure out that if the attrtimeo is N, then the attribute cache is valid from time, T, to T + N - 1, not T + N. Thus, the current attribute cache implementation is off by one because the attribute cache should expire at time, T + N. The time_in_range() macro was handy and looked right, but wasn't quite right for the desired semantics.

Adding tests to check to see if b and c are equal is tuning for the wrong case, I think. I believe that the majority of file systems are not mounted with "noac" or "actimeo=0", so the extra test would just be overhead for the common case.

Quoted 14 lines

Quoted 12 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +8 -0

+        *
+        * Otherwise, just check to see whether a is in [b, c).
+        */
+       if (c < b)
+               return time_after_eq(a, b) || time_before(a, c);
+       return time_after_eq(a, b) && time_before(a, c);
+}
+
 static inline int NFS_STALE(const struct inode *inode)

 {
      return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);

Quoted 130 lines · Peter Staubach

Quoted 4 lines · Chuck Lever

Hi Peter-

Hi, Chuck.

Quoted 30 lines

Quoted 26 lines · Peter Staubach

Hi.

I've been looking at a bugzilla which describes a problem where a customer was advised to use either the "noac" or "actimeo=0" mount options to solve a consistency problem that they were seeing in the file attributes. It turned out that this solution did not work reliably for them because sometimes, the local attribute cache was believed to be valid and not timed out. (With an attribute cache timeout of 0, the cache should always appear to be timed out.)

In looking at this situation, it appears to me that the problem is that the attribute cache timeout code has an off-by-one error in it. It is assuming that the cache is valid in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo]. The cache should be considered valid only in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo). With this change, the options, "noac" and "actimeo=0", work as originally expected.

While I was there, I addressed a problem with the jiffies overflowing on 32 bit systems. When overflow occurs, the value of read_cache_jiffies + attrtimeo can be less then the value of read_cache_jiffies. This would cause an unnecessary GETATTR over the wire.

Quoted 9 lines

Quoted 5 lines

Thoughts and/or comments? This is an updated patch which includes the previous support which was added to correct the noac/actimeo=0 handling.

A couple of random thoughts below.

Some thoughts in response --

Quoted 67 lines

Quoted 67 lines

Signed-off-by Peter Staubach <[email protected]>

fs/nfs/dir.c renamed from fs/nfs/dir.c.org +1 -1

@@ -1808,7 +1808,7 @@ static int nfs_access_get_cached(struct
      cache = nfs_access_search_rbtree(inode, cred);
      if (cache == NULL)
              goto out;
-       if (!time_in_range(jiffies, cache->jiffies, cache->jiffies +
 nfsi->attrtimeo))
+       if (!nfs_time_in_range_open(jiffies, cache->jiffies,

cache->jiffies
+ nfsi->attrtimeo))

              goto out_stale;
      res->jiffies = cache->jiffies;
      res->cred = cache->cred;

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -2

@@ -706,14 +706,7 @@ int nfs_attribute_timeout(struct inode *
 
      if (nfs_have_delegation(inode, FMODE_READ))
              return 0;
-       /*
-        * Special case: if the attribute timeout is set to 0, then

always
- * treat the cache as having expired (unless
holding

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -5

-        *               a delegation).
-        */
-       if (nfsi->attrtimeo == 0)
-               return 1;
-       return !time_in_range(jiffies, nfsi->read_cache_jiffies,

nfsi->read_cache_jiffies + nfsi->attrtimeo);
+ return !nfs_time_in_range_open(jiffies, nfsi->read_cache_jiffies,
nfsi->read_cache_jiffies + nfsi->attrtimeo);
}

/**

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -1

@@ -1098,7 +1091,7 @@ static int nfs_update_inode(struct inode
              nfsi->attrtimeo_timestamp = now;
              nfsi->last_updated = now;
      } else {
-               if (!time_in_range(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
+               if (!nfs_time_in_range_open(now,
 nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {

                      if ((nfsi->attrtimeo <<= 1) >
NFS_MAXATTRTIMEO(inode))
                              nfsi->attrtimeo = NFS_MAXATTRTIMEO(inode);
                      nfsi->attrtimeo_timestamp = now;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +1 -1

@@ -121,7 +121,7 @@ struct nfs_inode {
       *
       * We need to revalidate the cached attrs for this inode if
       *
-        *      jiffies - read_cache_jiffies > attrtimeo
+        *      jiffies - read_cache_jiffies >= attrtimeo
       */
      unsigned long           read_cache_jiffies;
      unsigned long           attrtimeo;

@@ -244,6 +244,22 @@ static inline unsigned NFS_MAXATTRTIMEO(
      return S_ISDIR(inode->i_mode) ? nfss->acdirmax : nfss->acregmax;
 }
 

Quoted 12 lines

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 4 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +2 -0

+static inline int nfs_time_in_range_open(unsigned long a,
+                               unsigned long b, unsigned long c)
 
 

All of nfs_time_in_range_open's callers use a sum of 'b' and 'nfsi->attrtimeo' for 'c'. Would it be cleaner to pass in nfsi->attrtimeo' rather than 'b + nfsi->attrtimeo' and do the sum here? It might make sense to explicitly check nfsi->attrtimeo for zero here to document the special behavior of "actimeo=0".

Quoted 3 lines

The behavior of "actimeo=0" isn't any more special than "actimeo=1". It simply indicates that the attribute timeout is 0 jiffies long.

Right. I'm simply suggesting that adding explicit code is good documentation for this case. It calls it out so developers remember that to check that case when they change this code.

You are correct that "noac/actimeo=0" is not the common case; however, it is a case that gets ignored and therefore broken easily, and that usually results in corruption of a customer's data.

Quoted 34 lines

I thought about reducing the arguments, but it didn't seem to yield anything any clearer to me.

Quoted 24 lines

Alternately, checking explicitly if b and c are equal might accomplish the same without changing the synopsis.

Also, all of nfs_time_in_range_open's callers negate the return value.
Would the code in the callers be cleaner if that negation was moved
into nfs_time_in_range_open? You might rename nfs_time_in_range_open() as nfs_cache_has_expired(), for example, to make the 'if' statements in the callers make sense in English.

My feeling is that if you have to sit and stare at this for 5 minutes to understand precisely how it works, it has already become too obfuscated. In addition to fixing any bugs, I wonder if we can make it easier to understand and maintain as well.

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO(

Quoted 8 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +6 -0

+{
+       /*
+        * If c is less then b, then the jiffies have wrapped.
+        * If so, then check to see if a is between b and the
+        * max jiffies value or between 0 and the value of c.
+        * This is the range between b and c.
 

Quoted 6 lines

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

Quoted 8 lines

Clearly, time_in_range() is not sufficient if the 'c' has wrapped. It only tests to see if a >=b and a <= c. If 'c' is less than 'b', then time_in_range() will return false.

I am reluctant to fix time_in_range() because I don't know that it is broken. It appears to me that it works for other uses, because otherwise, someone would have "fixed" it.

The only callers I found are the NFS client and the RPC client's auth cache, so it is probably safe to change time_in_range() without concern for breaking someone else's code. It's all ours, baby :-)

<[email protected]> introduced time_in_range() a year ago with commit c7e15961 for, it appears from his patch description, very similar reasons to your fix. It might be a good idea to discuss the wrapping bug with him.

Quoted 21 lines

Quoted 14 lines

You could then simplify this to "return b != c && time_in_range(a, b, c);" or something like that. Or if you negate the return value here:

static inline nfs_attributes_have_expired(unsigned long current,
unsigned long
start, unsigned long end)

{
       return (start == end) || !time_in_range(current, start, end);
}

My 0.02USD.

The change, which makes attrtimeo=0 work for free, is to figure out that if the attrtimeo is N, then the attribute cache is valid from time, T, to T + N - 1, not T + N. Thus, the current attribute cache implementation is off by one because the attribute cache should expire at time, T + N. The time_in_range() macro was handy and looked right, but wasn't quite right for the desired semantics.

Quoted 5 lines

Adding tests to check to see if b and c are equal is tuning for the wrong case, I think. I believe that the majority of file systems are not mounted with "noac" or "actimeo=0", so the extra test would just be overhead for the common case.

True enough, but you can "fix" that simply by reversing the two checks:

   return !time_in_range(a, b, c) || unlikely(b == c);

Again, I think there is some value in explicitly documenting the actimeo=0 case here whether or not it is covered implicitly by time_in_range(), precisely because it is not the common case and is often forgotten when changing attribute cache-related logic. This is exactly why we are now here fixing this problem!

The comments you added here nicely explain the complexity of the time checks, but do not explicitly state that actimeo=0 must work after any changes to this code -- one of the important reasons that you have open-coded the time comparisons rather than reusing time_in_range().

For me this is one of those times where cleverly folding all the cases into a single group of logic makes the code less good because it increases the chances of breakage later on, for example if time_in_range() is changed by someone else who doesn't have local knowledge of NFS.

Quoted 15 lines

Quoted 14 lines

Quoted 13 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +8 -0

+        *
+        * Otherwise, just check to see whether a is in [b, c).
+        */
+       if (c < b)
+               return time_after_eq(a, b) || time_before(a, c);
+       return time_after_eq(a, b) && time_before(a, c);
+}
+

static inline int NFS_STALE(const struct inode *inode)

 {
      return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);

Quoted 260 lines · Chuck Lever

Quoted 141 lines · Peter Staubach

Quoted 5 lines · Chuck Lever

Hi Peter-

Hi, Chuck.

Quoted 32 lines

Quoted 27 lines · Peter Staubach

Hi.

I've been looking at a bugzilla which describes a problem where a customer was advised to use either the "noac" or "actimeo=0" mount options to solve a consistency problem that they were seeing in the file attributes. It turned out that this solution did not work reliably for them because sometimes, the local attribute cache was believed to be valid and not timed out. (With an attribute cache timeout of 0, the cache should always appear to be timed out.)

In looking at this situation, it appears to me that the problem is that the attribute cache timeout code has an off-by-one error in it. It is assuming that the cache is valid in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo]. The cache should be considered valid only in the region, [read_cache_jiffies, read_cache_jiffies + attrtimeo). With this change, the options, "noac" and "actimeo=0", work as originally expected.

While I was there, I addressed a problem with the jiffies overflowing on 32 bit systems. When overflow occurs, the value of read_cache_jiffies + attrtimeo can be less then the value of read_cache_jiffies. This would cause an unnecessary GETATTR over the wire.

Quoted 10 lines

Quoted 5 lines

Thoughts and/or comments? This is an updated patch which includes the previous support which was added to correct the noac/actimeo=0 handling.

A couple of random thoughts below.

Some thoughts in response --

Quoted 69 lines

Quoted 69 lines

Signed-off-by Peter Staubach <[email protected]>

fs/nfs/dir.c renamed from fs/nfs/dir.c.org +1 -1

@@ -1808,7 +1808,7 @@ static int nfs_access_get_cached(struct
     cache = nfs_access_search_rbtree(inode, cred);
     if (cache == NULL)
             goto out;
-       if (!time_in_range(jiffies, cache->jiffies, cache->jiffies +
 nfsi->attrtimeo))
+       if (!nfs_time_in_range_open(jiffies, cache->jiffies,

cache->jiffies
+ nfsi->attrtimeo))

             goto out_stale;
     res->jiffies = cache->jiffies;
     res->cred = cache->cred;

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -2

@@ -706,14 +706,7 @@ int nfs_attribute_timeout(struct inode *
 
     if (nfs_have_delegation(inode, FMODE_READ))
             return 0;
-       /*
-        * Special case: if the attribute timeout is set to 0, then

always
- * treat the cache as having expired (unless
holding

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +0 -5

-        *               a delegation).
-        */
-       if (nfsi->attrtimeo == 0)
-               return 1;
-       return !time_in_range(jiffies, nfsi->read_cache_jiffies,

nfsi->read_cache_jiffies + nfsi->attrtimeo);
+ return !nfs_time_in_range_open(jiffies,
nfsi->read_cache_jiffies,
nfsi->read_cache_jiffies + nfsi->attrtimeo);
}

/**

fs/nfs/inode.c renamed from fs/nfs/inode.c.org +1 -1

@@ -1098,7 +1091,7 @@ static int nfs_update_inode(struct inode
             nfsi->attrtimeo_timestamp = now;
             nfsi->last_updated = now;
     } else {
-               if (!time_in_range(now, nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {
+               if (!nfs_time_in_range_open(now,
 nfsi->attrtimeo_timestamp,
 nfsi->attrtimeo_timestamp + nfsi->attrtimeo)) {

                     if ((nfsi->attrtimeo <<= 1) >
NFS_MAXATTRTIMEO(inode))
                             nfsi->attrtimeo = NFS_MAXATTRTIMEO(inode);
                     nfsi->attrtimeo_timestamp = now;

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +1 -1

@@ -121,7 +121,7 @@ struct nfs_inode {
      *
      * We need to revalidate the cached attrs for this inode if
      *
-        *      jiffies - read_cache_jiffies > attrtimeo
+        *      jiffies - read_cache_jiffies >= attrtimeo
      */
     unsigned long           read_cache_jiffies;
     unsigned long           attrtimeo;

@@ -244,6 +244,22 @@ static inline unsigned NFS_MAXATTRTIMEO(
     return S_ISDIR(inode->i_mode) ? nfss->acdirmax : nfss->acregmax;
 }
 

Quoted 13 lines

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO( · line 250

Quoted 4 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +2 -0

+static inline int nfs_time_in_range_open(unsigned long a,
+                               unsigned long b, unsigned long c)
 
 

All of nfs_time_in_range_open's callers use a sum of 'b' and 'nfsi->attrtimeo' for 'c'. Would it be cleaner to pass in nfsi->attrtimeo' rather than 'b + nfsi->attrtimeo' and do the sum here? It might make sense to explicitly check nfsi->attrtimeo for zero here to document the special behavior of "actimeo=0".

Quoted 3 lines

The behavior of "actimeo=0" isn't any more special than "actimeo=1". It simply indicates that the attribute timeout is 0 jiffies long.

Right. I'm simply suggesting that adding explicit code is good documentation for this case. It calls it out so developers remember that to check that case when they change this code.

You are correct that "noac/actimeo=0" is not the common case; however, it is a case that gets ignored and therefore broken easily, and that usually results in corruption of a customer's data.

Quoted 39 lines

I thought about reducing the arguments, but it didn't seem to yield anything any clearer to me.

Quoted 27 lines

Alternately, checking explicitly if b and c are equal might accomplish the same without changing the synopsis.

Also, all of nfs_time_in_range_open's callers negate the return value.
Would the code in the callers be cleaner if that negation was moved
into nfs_time_in_range_open? You might rename nfs_time_in_range_open() as nfs_cache_has_expired(), for example, to make the 'if' statements in the callers make sense in English.

My feeling is that if you have to sit and stare at this for 5 minutes to understand precisely how it works, it has already become too obfuscated. In addition to fixing any bugs, I wonder if we can make it easier to understand and maintain as well.

include/linux/nfs_fs.h static inline unsigned NFS_MAXATTRTIMEO( · line 257

Quoted 9 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +6 -0

+{
+       /*
+        * If c is less then b, then the jiffies have wrapped.
+        * If so, then check to see if a is between b and the
+        * max jiffies value or between 0 and the value of c.
+        * This is the range between b and c.
 

Quoted 6 lines

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

Quoted 8 lines

Clearly, time_in_range() is not sufficient if the 'c' has wrapped. It only tests to see if a >=b and a <= c. If 'c' is less than 'b', then time_in_range() will return false.

I am reluctant to fix time_in_range() because I don't know that it is broken. It appears to me that it works for other uses, because otherwise, someone would have "fixed" it.

The only callers I found are the NFS client and the RPC client's auth cache, so it is probably safe to change time_in_range() without concern for breaking someone else's code. It's all ours, baby :-)

<[email protected]> introduced time_in_range() a year ago with commit c7e15961 for, it appears from his patch description, very similar reasons to your fix. It might be a good idea to discuss the wrapping bug with him.

Quoted 23 lines

Quoted 4 lines

You could then simplify this to "return b != c && time_in_range(a, b, c);" or something like that. Or if you negate the return value here:

Quoted 11 lines

static inline nfs_attributes_have_expired(unsigned long current,
unsigned long
start, unsigned long end)

{
      return (start == end) || !time_in_range(current, start, end);
}

My 0.02USD.

The change, which makes attrtimeo=0 work for free, is to figure out that if the attrtimeo is N, then the attribute cache is valid from time, T, to T + N - 1, not T + N. Thus, the current attribute cache implementation is off by one because the attribute cache should expire at time, T + N. The time_in_range() macro was handy and looked right, but wasn't quite right for the desired semantics.

Quoted 5 lines

Adding tests to check to see if b and c are equal is tuning for the wrong case, I think. I believe that the majority of file systems are not mounted with "noac" or "actimeo=0", so the extra test would just be overhead for the common case.

True enough, but you can "fix" that simply by reversing the two checks:

  return !time_in_range(a, b, c) || unlikely(b == c);

Again, I think there is some value in explicitly documenting the actimeo=0 case here whether or not it is covered implicitly by time_in_range(), precisely because it is not the common case and is often forgotten when changing attribute cache-related logic. This is exactly why we are now here fixing this problem!

The comments you added here nicely explain the complexity of the time checks, but do not explicitly state that actimeo=0 must work after any changes to this code -- one of the important reasons that you have open-coded the time comparisons rather than reusing time_in_range().

Quoted 7 lines

For me this is one of those times where cleverly folding all the cases into a single group of logic makes the code less good because it increases the chances of breakage later on, for example if time_in_range() is changed by someone else who doesn't have local knowledge of NFS.

This was really just an off by one bug. _All_ attribute cache timeouts are one clock tick too long.

Adding unlikely() around the test may help to reduce its cost, but I don't think that it will make it zero cost. Ordering the tests will also help to minimize the cost, but still won't make the additional test zero cost.

Actually, the _only_ reason that I implemented nfs_time_in_range_open instead of just modifying time_in_range() was that I didn't want to impact things that were orthogonal to the bug that I needed to fix. Given that time_in_range() is only used by NFS and RPC, perhaps we can safely modify it. If time_in_range had only been being used by those three tests, I would have simply updated it.

Simply correcting the math gets us the desired functionality for zero additional cost over the broken support. In my viewpoint, it is also the easiest to understand because there won't be any special cases to worry about and the math will match the desired semantics.

Quoted 21 lines

Quoted 17 lines

Quoted 15 lines

Quoted 13 lines

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.org +8 -0

+        *
+        * Otherwise, just check to see whether a is in [b, c).
+        */
+       if (c < b)
+               return time_after_eq(a, b) || time_before(a, c);
+       return time_after_eq(a, b) && time_before(a, c);
+}
+

static inline int NFS_STALE(const struct inode *inode)

 {
     return test_bit(NFS_INO_STALE, &NFS_I(inode)->flags);

Quoted 5 lines

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

Clearly, time_in_range() is not sufficient if the 'c' has wrapped. It only tests to see if a >=b and a <= c. If 'c' is less than 'b', then time_in_range() will return false.

The change, which makes attrtimeo=0 work for free, is to figure out that if the attrtimeo is N, then the attribute cache is valid from time, T, to T + N - 1, not T + N. Thus, the current attribute cache implementation is off by one because the attribute cache should expire at time, T + N. The time_in_range() macro was handy and looked right, but wasn't quite right for the desired semantics.

Adding tests to check to see if b and c are equal is tuning for the wrong case, I think. I believe that the majority of file systems are not mounted with "noac" or "actimeo=0", so the extra test would just be overhead for the common case.

Quoted 10 lines · Peter Staubach

Quoted 6 lines

include/linux/jiffies.h claims it handles jiffy wrapping correctly. Why isn't time_in_range() sufficient if 'c' has wrapped? If it isn't, should you fix time_in_range() too?

Clearly, time_in_range() is not sufficient if the 'c' has wrapped. It only tests to see if a >=b and a <= c. If 'c' is less than 'b', then time_in_range() will return false.

Hmm... The actual test in the current time_in_range() should be

    ((long)b - (long)a) <= 0) && ((long)a - (long)c) <= 0

Which is _not_ the same as testing for a>=b && a<=c in the case of a sign wrap. Can you show me a case where we might have a problem?

The only case I can think of is if

    ((long) c - (long) b) < 0

(IOW: if the range itself is too large to fit into a signed long). I can't imagine that we will ever find ourselves in that situation.

Quoted 7 lines

The change, which makes attrtimeo=0 work for free, is to figure out that if the attrtimeo is N, then the attribute cache is valid from time, T, to T + N - 1, not T + N. Thus, the current attribute cache implementation is off by one because the attribute cache should expire at time, T + N. The time_in_range() macro was handy and looked right, but wasn't quite right for the desired semantics.

Quoted 5 lines

Adding tests to check to see if b and c are equal is tuning for the wrong case, I think. I believe that the majority of file systems are not mounted with "noac" or "actimeo=0", so the extra test would just be overhead for the common case.

I agree with this.

Given that we seem to "own" time_in_range(), how about the attached patch which just modifies time_in_range() to calculate [b,c) instead of [b,c], removes the special case for attrtimeo==0 in nfs_attribute_timeout() and adds a comment that Chuck requested concerning the need to ensure that zero timeout values continue to work?

Quoted 7 lines · Peter Staubach

Given that we seem to "own" time_in_range(), how about the attached patch which just modifies time_in_range() to calculate [b,c) instead of [b,c], removes the special case for attrtimeo==0 in nfs_attribute_timeout() and adds a comment that Chuck requested concerning the need to ensure that zero timeout values continue to work?

I'm fine with that, but still suggest that you change the name in order to avoid confusing possible out-of-tree users.