Skip to contentSkip to conversation

Should we be aggressively invalidating cache when using -onolock?

PATCH Sachin Prabhu 5 participants 9 messages Sep 17, 2010 – Oct 20, 2010 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1103741.22.1284726314119.JavaMail.sprabhu@dhcp-1-233.fab.redhat.com

#1 lore raw

We came across an issue where the performance of an application using flocks on RHEL 4(2.6.9 kernel) was far better when compared to the performance of the same application on RHEL 5(2.6.18 kernel). The nfs client behavior when performing flocks on RHEL 4 and RHEL 5 differ. To ensure we had a level playing field, we repeated the tests using the mount option -o nolock.

The performance on RHEL 5 improved slightly but was still pretty bad when compared to performance on RHEL 4. On closer observation, it was seen that there are a large number of READ requests on RHEL 5 while on RHEL 4 there were hardly any. This difference in behavior was caused by the code which invalidates the cache in the do_setlk() function and results in the RHEL 5 client performing a large number of READ requests.

In this case, the files were only being accessed by one client. This is why the nolock mount option was used. When running such workloads, the aggressive invalidation of the cache is unnecessary. This patch improves the performance in such a scenario.

Is this a good idea?

The patch will need to be respinned to accomodate Suresh Jayaraman's patch introducing '-olocal_lock'.

Sachin Prabhu

nfs: Skip zapping caches when using -o nolock

When using -onolock, it is assumed that the file will not be accessed/modified from multiple sources. In such cases, aggressive invalidation of cache is not required.

fs/nfs/file.c +20 -17
index eb51bd6..bfd9c1a 100644
@@ -733,24 +733,22 @@ static int do_vfs_lock(struct file *file, struct file_lock *fl)
 static int do_unlk(struct file *filp, int cmd, struct file_lock *fl)
 {
 	struct inode *inode = filp->f_mapping->host;
-	int status;
+
+	/* NOTE: special case
+	 * If we're signalled while cleaning up locks on process exit, we
+	 * still need to complete the unlock.
+	 */
+
+	/* Use local locking if mounted with "-onolock" */
+	if (NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM)
+		return do_vfs_lock(filp, fl);
 
 	/*
 	 * Flush all pending writes before doing anything
 	 * with locks..
 	 */
 	nfs_sync_mapping(filp->f_mapping);
-
-	/* NOTE: special case
-	 * 	If we're signalled while cleaning up locks on process exit, we
-	 * 	still need to complete the unlock.
-	 */
-	/* Use local locking if mounted with "-onolock" */
-	if (!(NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM))
-		status = NFS_PROTO(inode)->lock(filp, cmd, fl);
-	else
-		status = do_vfs_lock(filp, fl);
-	return status;
+	return NFS_PROTO(inode)->lock(filp, cmd, fl);
 }
 
 static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
@@ -759,6 +757,15 @@ static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
 	int status;
 
 	/*
+	 * Use local locking and skip cache writeback or invalidation
+	 * if mounted with "-onolock"
+	 */
+	if (NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM) {
+		status = do_vfs_lock(filp, fl);
+		goto out;
+	}
+
+	/*
 	 * Flush all pending writes before doing anything
 	 * with locks..
 	 */
@@ -766,11 +773,7 @@ static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
 	if (status != 0)
 		goto out;
 
-	/* Use local locking if mounted with "-onolock" */
-	if (!(NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM))
-		status = NFS_PROTO(inode)->lock(filp, cmd, fl);
-	else
-		status = do_vfs_lock(filp, fl);
+	status = NFS_PROTO(inode)->lock(filp, cmd, fl);
 	if (status < 0)
 		goto out;
 	/*
#2 lore raw
Quoted 5 lines · Sachin Prabhu

We came across an issue where the performance of an application using flocks on RHEL 4(2.6.9 kernel) was far better when compared to the performance of the same application on RHEL 5(2.6.18 kernel). The nfs client behavior when performing flocks on RHEL 4 and RHEL 5 differ. To ensure we had a level playing field, we repeated the tests using the mount option -o nolock.

The performance on RHEL 5 improved slightly but was still pretty bad when compared to performance on RHEL 4. On closer observation, it was seen that there are a large number of READ requests on RHEL 5 while on RHEL 4 there were hardly any. This difference in behavior was caused by the code which invalidates the cache in the do_setlk() function and results in the RHEL 5 client performing a large number of READ requests.

In this case, the files were only being accessed by one client. This is why the nolock mount option was used. When running such workloads, the aggressive invalidation of the cache is unnecessary. This patch improves the performance in such a scenario.

Makes sense to me.

(Is it possible that somebody might depend on lock/unlock to keep their meaning of "invalidate cache/flush changes" even when they don't care bout checking for inter-client lock conflicts? That sounds like an odd use case to me.)

--b.

Quoted 81 lines

Is this a good idea?

The patch will need to be respinned to accomodate Suresh Jayaraman's patch introducing '-olocal_lock'.

Sachin Prabhu

nfs: Skip zapping caches when using -o nolock

When using -onolock, it is assumed that the file will not be accessed/modified from multiple sources. In such cases, aggressive invalidation of cache is not required.

fs/nfs/file.c +20 -17
index eb51bd6..bfd9c1a 100644
@@ -733,24 +733,22 @@ static int do_vfs_lock(struct file *file, struct file_lock *fl)
 static int do_unlk(struct file *filp, int cmd, struct file_lock *fl)
 {
 	struct inode *inode = filp->f_mapping->host;
-	int status;
+
+	/* NOTE: special case
+	 * If we're signalled while cleaning up locks on process exit, we
+	 * still need to complete the unlock.
+	 */
+
+	/* Use local locking if mounted with "-onolock" */
+	if (NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM)
+		return do_vfs_lock(filp, fl);
 
 	/*
 	 * Flush all pending writes before doing anything
 	 * with locks..
 	 */
 	nfs_sync_mapping(filp->f_mapping);
-
-	/* NOTE: special case
-	 * 	If we're signalled while cleaning up locks on process exit, we
-	 * 	still need to complete the unlock.
-	 */
-	/* Use local locking if mounted with "-onolock" */
-	if (!(NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM))
-		status = NFS_PROTO(inode)->lock(filp, cmd, fl);
-	else
-		status = do_vfs_lock(filp, fl);
-	return status;
+	return NFS_PROTO(inode)->lock(filp, cmd, fl);
 }
 
 static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
@@ -759,6 +757,15 @@ static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
 	int status;
 
 	/*
+	 * Use local locking and skip cache writeback or invalidation
+	 * if mounted with "-onolock"
+	 */
+	if (NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM) {
+		status = do_vfs_lock(filp, fl);
+		goto out;
+	}
+
+	/*
 	 * Flush all pending writes before doing anything
 	 * with locks..
 	 */
@@ -766,11 +773,7 @@ static int do_setlk(struct file *filp, int cmd, struct file_lock *fl)
 	if (status != 0)
 		goto out;
 
-	/* Use local locking if mounted with "-onolock" */
-	if (!(NFS_SERVER(inode)->flags & NFS_MOUNT_NONLM))
-		status = NFS_PROTO(inode)->lock(filp, cmd, fl);
-	else
-		status = do_vfs_lock(filp, fl);
+	status = NFS_PROTO(inode)->lock(filp, cmd, fl);
 	if (status < 0)
 		goto out;
 	/*
#3 lore raw
Quoted 9 lines · On Fri, 17 Sep 2010 13:46:44 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Sachin Prabhu

We came across an issue where the performance of an application using flocks on RHEL 4(2.6.9 kernel) was far better when compared to the performance of the same application on RHEL 5(2.6.18 kernel). The nfs client behavior when performing flocks on RHEL 4 and RHEL 5 differ. To ensure we had a level playing field, we repeated the tests using the mount option -o nolock.

The performance on RHEL 5 improved slightly but was still pretty bad when compared to performance on RHEL 4. On closer observation, it was seen that there are a large number of READ requests on RHEL 5 while on RHEL 4 there were hardly any. This difference in behavior was caused by the code which invalidates the cache in the do_setlk() function and results in the RHEL 5 client performing a large number of READ requests.

In this case, the files were only being accessed by one client. This is why the nolock mount option was used. When running such workloads, the aggressive invalidation of the cache is unnecessary. This patch improves the performance in such a scenario.

Makes sense to me.

Agreed. This is potentially a huge performance win for some workloads.

Quoted 5 lines

(Is it possible that somebody might depend on lock/unlock to keep their meaning of "invalidate cache/flush changes" even when they don't care bout checking for inter-client lock conflicts? That sounds like an odd use case to me.)

Aye, there's the rub. It might cause anyone doing this to regress. My gut feeling is that those people are a minority if any exist at all. The consequences of such a change for them could be pretty ugly but I'm not sure we owe them any consistency guarantees in such a case.

Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw
Quoted 21 lines · Jeff Layton
Quoted 9 lines · On Fri, 17 Sep 2010 13:46:44 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Sachin Prabhu

We came across an issue where the performance of an application using flocks on RHEL 4(2.6.9 kernel) was far better when compared to the performance of the same application on RHEL 5(2.6.18 kernel). The nfs client behavior when performing flocks on RHEL 4 and RHEL 5 differ. To ensure we had a level playing field, we repeated the tests using the mount option -o nolock.

The performance on RHEL 5 improved slightly but was still pretty bad when compared to performance on RHEL 4. On closer observation, it was seen that there are a large number of READ requests on RHEL 5 while on RHEL 4 there were hardly any. This difference in behavior was caused by the code which invalidates the cache in the do_setlk() function and results in the RHEL 5 client performing a large number of READ requests.

In this case, the files were only being accessed by one client. This is why the nolock mount option was used. When running such workloads, the aggressive invalidation of the cache is unnecessary. This patch improves the performance in such a scenario.

Makes sense to me.

Agreed. This is potentially a huge performance win for some workloads.

Quoted 5 lines

(Is it possible that somebody might depend on lock/unlock to keep their meaning of "invalidate cache/flush changes" even when they don't care bout checking for inter-client lock conflicts? That sounds like an odd use case to me.)

Quoted 4 lines

Aye, there's the rub. It might cause anyone doing this to regress. My gut feeling is that those people are a minority if any exist at all. The consequences of such a change for them could be pretty ugly but I'm not sure we owe them any consistency guarantees in such a case.

Yeah, I haven't seen any documentation of the revalidate-caches-but-don't-lock mode that has been accidentally implemented here, and I don't think anyone could sensibly depend on it.

--b.

#5 lore raw
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Jeff Layton
Quoted 9 lines · On Fri, 17 Sep 2010 13:46:44 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Sachin Prabhu

We came across an issue where the performance of an application using flocks on RHEL 4(2.6.9 kernel) was far better when compared to the performance of the same application on RHEL 5(2.6.18 kernel). The nfs client behavior when performing flocks on RHEL 4 and RHEL 5 differ. To ensure we had a level playing field, we repeated the tests using the mount option -o nolock.

The performance on RHEL 5 improved slightly but was still pretty bad when compared to performance on RHEL 4. On closer observation, it was seen that there are a large number of READ requests on RHEL 5 while on RHEL 4 there were hardly any. This difference in behavior was caused by the code which invalidates the cache in the do_setlk() function and results in the RHEL 5 client performing a large number of READ requests.

In this case, the files were only being accessed by one client. This is why the nolock mount option was used. When running such workloads, the aggressive invalidation of the cache is unnecessary. This patch improves the performance in such a scenario.

Makes sense to me.

Agreed. This is potentially a huge performance win for some workloads.

Quoted 5 lines

(Is it possible that somebody might depend on lock/unlock to keep their meaning of "invalidate cache/flush changes" even when they don't care bout checking for inter-client lock conflicts? That sounds like an odd use case to me.)

Quoted 8 lines
Quoted 4 lines

Aye, there's the rub. It might cause anyone doing this to regress. My gut feeling is that those people are a minority if any exist at all. The consequences of such a change for them could be pretty ugly but I'm not sure we owe them any consistency guarantees in such a case.

Yeah, I haven't seen any documentation of the revalidate-caches-but-don't-lock mode that has been accidentally implemented here, and I don't think anyone could sensibly depend on it.

At one point long ago, I had asked Trond if we could get rid of the cache-invalidation-on-lock behavior if "-onolock" was in effect. He said at the time that this would eliminate the only recourse applications have for invalidating the data cache in case it was stale, and NACK'd the request.

I suggested introducing a new mount option called "llock" that would be semantically the same as "llock" on other operating systems, to do this. It never went anywhere.

We now seem to have a fresh opportunity to address this issue with the recent addition of "local_lock". Can we augment this option or add another which allows better control of caching behavior during a file lock?

It also seems to me that if RHEL 4 is _not_ invalidating on lock, then it is not working as designed. AFAIK the Linux NFS client has always invalidated a file's data cache on lock. Did I misread something?

Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
#6 lore raw
Quoted 5 lines · Chuck Lever

At one point long ago, I had asked Trond if we could get rid of the cache-invalidation-on-lock behavior if "-onolock" was in effect. He said at the time that this would eliminate the only recourse applications have for invalidating the data cache in case it was stale, and NACK'd the request.

Argh. I guess I can see the argument, though.

Quoted 8 lines

I suggested introducing a new mount option called "llock" that would be semantically the same as "llock" on other operating systems, to do this. It never went anywhere.

We now seem to have a fresh opportunity to address this issue with the recent addition of "local_lock". Can we augment this option or add another which allows better control of caching behavior during a file lock?

I wouldn't stand in the way, but it does start to sound like a rather confusing array of choices.

--b.

Quoted 4 lines

It also seems to me that if RHEL 4 is _not_ invalidating on lock, then it is not working as designed. AFAIK the Linux NFS client has always invalidated a file's data cache on lock. Did I misread something?

#7 lore raw
Quoted 13 lines · On Mon, 20 Sep 2010 14:25:36 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Chuck Lever

At one point long ago, I had asked Trond if we could get rid of the cache-invalidation-on-lock behavior if "-onolock" was in effect. He said at the time that this would eliminate the only recourse applications have for invalidating the data cache in case it was stale, and NACK'd the request.

Argh. I guess I can see the argument, though.

Quoted 4 lines

I suggested introducing a new mount option called "llock" that would be semantically the same as "llock" on other operating systems, to do this. It never went anywhere.

Quoted 8 lines
Quoted 4 lines

We now seem to have a fresh opportunity to address this issue with the recent addition of "local_lock". Can we augment this option or add another which allows better control of caching behavior during a file lock?

I wouldn't stand in the way, but it does start to sound like a rather confusing array of choices.

I can sort of see the argument too, but on the other hand...does anyone *really* use locks in this way? If we want a mechanism to allow the client to force cache invalidation on an inode it seems like we'd be better off with an interface for that purpose only (dare I say ioctl? :).

Piggybacking this behavior into the locking interfaces seems like it punishes -o nolock performance for the benefit of some questionable usage patterns.

Mixing this in with -o local_lock also seems confusing, but if we want to do that it's probably best to make that call before any kernels ship with -o local_lock.

Trond, care to weigh in on this?

Signature · 1 line
Jeff Layton <[email protected]>
#8 lore raw
Quoted 31 lines · Jeff Layton
Quoted 13 lines · On Mon, 20 Sep 2010 14:25:36 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Chuck Lever

At one point long ago, I had asked Trond if we could get rid of the cache-invalidation-on-lock behavior if "-onolock" was in effect. He said at the time that this would eliminate the only recourse applications have for invalidating the data cache in case it was stale, and NACK'd the request.

Argh. I guess I can see the argument, though.

Quoted 4 lines

I suggested introducing a new mount option called "llock" that would be semantically the same as "llock" on other operating systems, to do this. It never went anywhere.

Quoted 8 lines
Quoted 4 lines

We now seem to have a fresh opportunity to address this issue with the recent addition of "local_lock". Can we augment this option or add another which allows better control of caching behavior during a file lock?

I wouldn't stand in the way, but it does start to sound like a rather confusing array of choices.

I can sort of see the argument too, but on the other hand...does anyone *really* use locks in this way? If we want a mechanism to allow the client to force cache invalidation on an inode it seems like we'd be better off with an interface for that purpose only (dare I say ioctl? :).

Quoted 4 lines

Piggybacking this behavior into the locking interfaces seems like it punishes -o nolock performance for the benefit of some questionable usage patterns.

+ 1

Quoted 1 line

Mixing this in with -o local_lock also seems confusing, but if we want

I too think it would be confusing and unwarranted. A separate interface would be a better choice IMHO..

Signature · 1 line
Suresh Jayaraman

A message that is not in the archive (sent off-list or not yet received): 20101005102752.67b75416-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#10 lore raw
Quoted 44 lines · ----- "Jeff Layton
Quoted 2 lines · On Mon, 20 Sep 2010 14:25:36 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Chuck Lever

At one point long ago, I had asked Trond if we could get rid of

the

Quoted 1 line
Quoted 1 line

cache-invalidation-on-lock behavior if "-onolock" was in effect.

He

Quoted 7 lines
Quoted 3 lines

said at the time that this would eliminate the only recourse applications have for invalidating the data cache in case it was stale, and NACK'd the request.

Argh. I guess I can see the argument, though.

Quoted 1 line

I suggested introducing a new mount option called "llock" that

would

Quoted 1 line
Quoted 1 line

be semantically the same as "llock" on other operating systems, to

do

Quoted 3 lines
Quoted 3 lines

this. It never went anywhere.

We now seem to have a fresh opportunity to address this issue with

the

Quoted 1 line
Quoted 1 line

recent addition of "local_lock". Can we augment this option or

add

Quoted 1 line
Quoted 1 line

another which allows better control of caching behavior during a

file

Quoted 3 lines
Quoted 1 line

lock?

I wouldn't stand in the way, but it does start to sound like a

rather

Quoted 2 lines

confusing array of choices.

I can sort of see the argument too, but on the other hand...does anyone
*really* use locks in this way? If we want a mechanism to allow the client to force cache invalidation on an inode it seems like we'd be better off with an interface for that purpose only (dare I say ioctl? :).

Piggybacking this behavior into the locking interfaces seems like it punishes -o nolock performance for the benefit of some questionable usage patterns.

Quoted 7 lines

Mixing this in with -o local_lock also seems confusing, but if we want
to do that it's probably best to make that call before any kernels ship
with -o local_lock.

Trond, care to weigh in on this?

Trond,

What do you think about this issue?

Sachin Prabhu