fix infinite loop in generic_file_splice_read()

generic_file_splice_read() goes into an infinite loop if it races with truncation. I've found this with fsx-linux on NFS over fuse.

Perhaps the whole while() loop is bogus, but I can't tell from a cursory glance at __generic_file_splice_read() if it will return zero only on EOF, or it can do that for other reasons as well. In the latter case the loop is obviously needed.

This simplistic patch fixes the issue for me.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/splice.c |   19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

Index: linux/fs/splice.c
===================================================================

fs/splice.c +10 -10

@@ -481,19 +481,20 @@ ssize_t generic_file_splice_read(struct
 {
 	ssize_t spliced;
 	int ret;
-	loff_t isize, left;
-
-	isize = i_size_read(in->f_mapping->host);
-	if (unlikely(*ppos >= isize))
-		return 0;
-
-	left = isize - *ppos;
-	if (unlikely(left < len))
-		len = left;
 
 	ret = 0;
 	spliced = 0;
 	while (len && !spliced) {
+		loff_t isize, left;
+
+		isize = i_size_read(in->f_mapping->host);
+		if (unlikely(*ppos >= isize))
+			return 0;
+
+		left = isize - *ppos;
+		if (unlikely(left < len))
+			len = left;
+
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);
 
 		if (ret < 0)
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

generic_file_splice_read() goes into an infinite loop if it races with truncation. I've found this with fsx-linux on NFS over fuse.

Perhaps the whole while() loop is bogus, but I can't tell from a cursory glance at __generic_file_splice_read() if it will return zero only on EOF, or it can do that for other reasons as well. In the latter case the loop is obviously needed.

This simplistic patch fixes the issue for me.

We found suspicious-looking code in generic_file_splice_read() back in February. See http://lkml.org/lkml/2008/2/29/443. I suspect that patch (if it works) will address the truncate lockup as well - it zaps the loop entirely.

Unfortunately Allard never got back to us (probably because he's running 2.6.24 which has a quite different generic_file_splice_read()) and the patch didn't get anywhere.

Here it is again.

It needs a changelog.

Nobody has tested this at all, to my knowledge.

Quoted 10 lines · On Wed, 09 Apr 2008 17:57:56 +0200 Miklos Szeredi <[email protected]> wrote:

generic_file_splice_read() goes into an infinite loop if it races with truncation. I've found this with fsx-linux on NFS over fuse.

Perhaps the whole while() loop is bogus, but I can't tell from a cursory glance at __generic_file_splice_read() if it will return zero only on EOF, or it can do that for other reasons as well. In the latter case the loop is obviously needed.

This simplistic patch fixes the issue for me.

We found suspicious-looking code in generic_file_splice_read() back in February. See http://lkml.org/lkml/2008/2/29/443. I suspect that patch (if it works) will address the truncate lockup as well - it zaps the loop entirely.

Unfortunately Allard never got back to us (probably because he's running 2.6.24 which has a quite different generic_file_splice_read()) and the patch didn't get anywhere.

Nobody has tested this at all, to my knowledge.

We found suspicious-looking code in generic_file_splice_read() back in February. See http://lkml.org/lkml/2008/2/29/443. I suspect that patch (if it works) will address the truncate lockup as well - it zaps the loop entirely.

Unfortunately Allard never got back to us (probably because he's running 2.6.24 which has a quite different generic_file_splice_read()) and the patch didn't get anywhere.

Another infinite loop, this one involving both client and server.

Basically what happens is that on the server nlm_fopen() calls nfsd_open() which returns -EACCES, to which nlm_fopen() returns NLM_LCK_DENIED.

On the client this will turn into a -EAGAIN (nlm_stat_to_errno()), which in will cause fcntl_setlk() to retry forever.

I _think_ the solution is to turn NLM_LCK_DENIED into ENOLCK for blocking locks, as NLM_LCK_BLOCKED is for the contended case. For testing the lock leave NLM_LCK_DENIED as EAGAIN. That still could be misleading, but at least there's no infinite loop in that case.

I've minimally tested this patch to verify that it cures the lockup, and that simple blocking locks keep working.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/lockd/clntproc.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/fs/lockd/clntproc.c
===================================================================

Quoted 27 lines · Miklos Szeredi

Another infinite loop, this one involving both client and server.

Basically what happens is that on the server nlm_fopen() calls nfsd_open() which returns -EACCES, to which nlm_fopen() returns NLM_LCK_DENIED.

On the client this will turn into a -EAGAIN (nlm_stat_to_errno()), which in will cause fcntl_setlk() to retry forever.

I _think_ the solution is to turn NLM_LCK_DENIED into ENOLCK for blocking locks, as NLM_LCK_BLOCKED is for the contended case. For testing the lock leave NLM_LCK_DENIED as EAGAIN. That still could be misleading, but at least there's no infinite loop in that case.

I've minimally tested this patch to verify that it cures the lockup, and that simple blocking locks keep working.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/lockd/clntproc.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/fs/lockd/clntproc.c
===================================================================

fs/lockd/clntproc.c again: · line 544

Quoted 9 lines

fs/lockd/clntproc.c +3 -0

@@ -536,6 +536,9 @@ again:
 		up_read(&host->h_rwsem);
 	}
 	status = nlm_stat_to_errno(resp->status);
+	/* Don't return EAGAIN, as that would make fcntl_setlk() loop */
+	if (status == -EAGAIN)
+		status = -ENOLCK;
 out_unblock:
 	nlmclnt_finish_block(block);
 	/* Cancel the blocked request if it is still pending */

Wait. There is something really weird going on here.

According to the spec, LCK_DENIED means 'the request failed' (i.e. ENOLCK is definitely correct)

OTOH, LCK_DENIED_NOLOCKS and LCK_DENIED_GRACE_PERIOD are both temporary failures, the first because the server had a resource problem, and the second because the server rebooted and is in the grace period (i.e. EAGAIN would appear to be more appropriate). See

http://www.opengroup.org/onlinepubs/9629799/chap10.htm#tagcjh_11_02_02_02

AFAICS, the correct thing to do is to fix nlm_stat_to_errno() by swapping the return values for NLM_LCK_DENIED and NLM_LCK_DENIED_NOLOCKS/NLM_LCK_DENIED_GRACE_PERIOD.

The problem is that there appears to be a similar confusion on the Linux server side in nlmsvc_lock(). :-(

Quoted 55 lines · Trond Myklebust

Quoted 27 lines · Miklos Szeredi

Another infinite loop, this one involving both client and server.

Basically what happens is that on the server nlm_fopen() calls nfsd_open() which returns -EACCES, to which nlm_fopen() returns NLM_LCK_DENIED.

On the client this will turn into a -EAGAIN (nlm_stat_to_errno()), which in will cause fcntl_setlk() to retry forever.

I _think_ the solution is to turn NLM_LCK_DENIED into ENOLCK for blocking locks, as NLM_LCK_BLOCKED is for the contended case. For testing the lock leave NLM_LCK_DENIED as EAGAIN. That still could be misleading, but at least there's no infinite loop in that case.

I've minimally tested this patch to verify that it cures the lockup, and that simple blocking locks keep working.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/lockd/clntproc.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/fs/lockd/clntproc.c
===================================================================

fs/lockd/clntproc.c again: · line 544

Quoted 9 lines

fs/lockd/clntproc.c +3 -0

@@ -536,6 +536,9 @@ again:
 		up_read(&host->h_rwsem);
 	}
 	status = nlm_stat_to_errno(resp->status);
+	/* Don't return EAGAIN, as that would make fcntl_setlk() loop */
+	if (status == -EAGAIN)
+		status = -ENOLCK;
 out_unblock:
 	nlmclnt_finish_block(block);
 	/* Cancel the blocked request if it is still pending */

Wait. There is something really weird going on here.

According to the spec, LCK_DENIED means 'the request failed' (i.e. ENOLCK is definitely correct)

OTOH, LCK_DENIED_NOLOCKS and LCK_DENIED_GRACE_PERIOD are both temporary failures, the first because the server had a resource problem, and the second because the server rebooted and is in the grace period (i.e. EAGAIN would appear to be more appropriate). See

http://www.opengroup.org/onlinepubs/9629799/chap10.htm#tagcjh_11_02_02_02

AFAICS, the correct thing to do is to fix nlm_stat_to_errno() by swapping the return values for NLM_LCK_DENIED and NLM_LCK_DENIED_NOLOCKS/NLM_LCK_DENIED_GRACE_PERIOD.

Quoted 2 lines

The problem is that there appears to be a similar confusion on the Linux server side in nlmsvc_lock(). :-(

Duh... Sorry, EAGAIN is indeed the correct return value for fcntl() when the lock attempt failed. I should have reread the manpage/posix spec before replying.

Quoted 56 lines · Trond Myklebust

Quoted 55 lines · Trond Myklebust

Quoted 27 lines · Miklos Szeredi

Another infinite loop, this one involving both client and server.

Basically what happens is that on the server nlm_fopen() calls nfsd_open() which returns -EACCES, to which nlm_fopen() returns NLM_LCK_DENIED.

On the client this will turn into a -EAGAIN (nlm_stat_to_errno()), which in will cause fcntl_setlk() to retry forever.

I _think_ the solution is to turn NLM_LCK_DENIED into ENOLCK for blocking locks, as NLM_LCK_BLOCKED is for the contended case. For testing the lock leave NLM_LCK_DENIED as EAGAIN. That still could be misleading, but at least there's no infinite loop in that case.

I've minimally tested this patch to verify that it cures the lockup, and that simple blocking locks keep working.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/lockd/clntproc.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/fs/lockd/clntproc.c
===================================================================

fs/lockd/clntproc.c again: · line 544

Quoted 9 lines

fs/lockd/clntproc.c +3 -0

@@ -536,6 +536,9 @@ again:
 		up_read(&host->h_rwsem);
 	}
 	status = nlm_stat_to_errno(resp->status);
+	/* Don't return EAGAIN, as that would make fcntl_setlk() loop */
+	if (status == -EAGAIN)
+		status = -ENOLCK;
 out_unblock:
 	nlmclnt_finish_block(block);
 	/* Cancel the blocked request if it is still pending */

Wait. There is something really weird going on here.

According to the spec, LCK_DENIED means 'the request failed' (i.e. ENOLCK is definitely correct)

OTOH, LCK_DENIED_NOLOCKS and LCK_DENIED_GRACE_PERIOD are both temporary failures, the first because the server had a resource problem, and the second because the server rebooted and is in the grace period (i.e. EAGAIN would appear to be more appropriate). See

http://www.opengroup.org/onlinepubs/9629799/chap10.htm#tagcjh_11_02_02_02

AFAICS, the correct thing to do is to fix nlm_stat_to_errno() by swapping the return values for NLM_LCK_DENIED and NLM_LCK_DENIED_NOLOCKS/NLM_LCK_DENIED_GRACE_PERIOD.

Quoted 6 lines

Quoted 2 lines

The problem is that there appears to be a similar confusion on the Linux server side in nlmsvc_lock(). :-(

Duh... Sorry, EAGAIN is indeed the correct return value for fcntl() when the lock attempt failed. I should have reread the manpage/posix spec before replying.

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Marc, Bruce, why was this
done, and how are filesystems now supposed to behave?

Quoted 6 lines

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Quoted 4 lines

Marc, Bruce, why was this
done, and how are filesystems now supposed to behave?

The assumption must have been that -EAGAIN could only mean that we needed to keep blocking, and hence was a nonsensical return from a filesystem lock method that waited itself for the lock to become available--such a method would return 0, -EINTR (or some more exotic error), or continue waiting.

Quoted 9 lines

Quoted 6 lines

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Yeah, that patch looks totally wrong. It's not generally a good idea to do a loop where the exit condition depends on something you don't control. And error values from filesystem methods are typically like that. For example with fuse, the error code could come from an unprivileged userspace process.

I didn't realize this aspect of the bug previously, because I concentrated on the lockd inconsistency.

Btw, why hasn't this work been posted on -fsdevel prior to merging into mainline?

Quoted 11 lines

Quoted 5 lines

Marc, Bruce, why was this
done, and how are filesystems now supposed to behave?

The assumption must have been that -EAGAIN could only mean that we needed to keep blocking, and hence was a nonsensical return from a filesystem lock method that waited itself for the lock to become available--such a method would return 0, -EINTR (or some more exotic error), or continue waiting.

EAGAIN for a blocking lock is nonsensical, so my original patch could still make sense. But that's no longer a regression, and not all that important.

Quoted 2 lines

Btw, why hasn't this work been posted on -fsdevel prior to merging into mainline?

http://marc.info/?l=linux-fsdevel&m=117581629326194&w=2

Quoted 9 lines

Quoted 6 lines

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Yeah, that patch looks totally wrong. It's not generally a good idea to do a loop where the exit condition depends on something you don't control. And error values from filesystem methods are typically like that. For example with fuse, the error code could come from an unprivileged userspace process.

I didn't realize this aspect of the bug previously, because I concentrated on the lockd inconsistency.

Quoted 16 lines

Quoted 9 lines

Quoted 6 lines

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Yeah, that patch looks totally wrong. It's not generally a good idea to do a loop where the exit condition depends on something you don't control. And error values from filesystem methods are typically like that. For example with fuse, the error code could come from an unprivileged userspace process.

Quoted 2 lines

I didn't realize this aspect of the bug previously, because I concentrated on the lockd inconsistency.

So, does this patch on its own fix the problem you saw?

Quoted 16 lines

Quoted 16 lines

Quoted 9 lines

Quoted 6 lines

OK. So the correct fix here should really be applied to fcntl_setlk(). There is absolutely no reason why we should be looping at all if the filesystem has a ->lock() method.

In fact, this looping behaviour was introduced recently in commit 7723ec9777d9832849b76475b1a21a2872a40d20.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Yeah, that patch looks totally wrong. It's not generally a good idea to do a loop where the exit condition depends on something you don't control. And error values from filesystem methods are typically like that. For example with fuse, the error code could come from an unprivileged userspace process.

Quoted 4 lines

Quoted 2 lines

I didn't realize this aspect of the bug previously, because I concentrated on the lockd inconsistency.

So, does this patch on its own fix the problem you saw?

Yes. With the patch applied, the test program returns "lockf:
Resource temporarily unavailable" instead of hanging.

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Quoted 2 lines

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Another complaint about this series: using EINPROGRESS to signal asynchronous locking looks really fishy. How does the filesystem know, that the caller wants to do async locking?

How do we make sure,
that the filesystem (like fuse or 9p, which "blindly" return the error from the server) doesn't return EINPROGRESS even when it's _not_ doing an asynchronous lock?

I think it would have been much cleaner to have a completely separate interface for async locking, instead of trying to cram that into f_op->lock().

Would that be possible to fix now? Or at least make EINPROGRESS a kernel-internal error value (>512), to make it that it has a special meaning for the _kernel only_?

Quoted 6 lines

Quoted 2 lines

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Another complaint about this series: using EINPROGRESS to signal asynchronous locking looks really fishy. How does the filesystem know, that the caller wants to do async locking?

The caller sets a fl_grant callback. But I guess the interesting question is how the caller knows that the filesystem is really going to return the results asynchronously:

Quoted 4 lines

How do we make sure,
that the filesystem (like fuse or 9p, which "blindly" return the error from the server) doesn't return EINPROGRESS even when it's _not_ doing an asynchronous lock?

Right, there's no safeguard there--if fuse returns EINPROGRESS, then we'll wait for a grant callback that's not going to come. It should time out, so that's not a total disaster, but still.

Anyway, I'm not sure what to do about that.

Quoted 4 lines

I think it would have been much cleaner to have a completely separate interface for async locking, instead of trying to cram that into f_op->lock().

Maybe so. ->lock() had quite a bit crammed into it even before this.

Quoted 3 lines

Would that be possible to fix now? Or at least make EINPROGRESS a kernel-internal error value (>512), to make it that it has a special meaning for the _kernel only_?

Perhaps so.

The behavior of lockd will still depend to some degree on the exact error returned from the filesystem--e.g. if you return -EAGAIN from ->lock() without later calling ->fl_grant() it will cause some unexpected delays. (Though again clients will eventually give up and poll for the lock.)

Quoted 21 lines · Miklos Szeredi

Quoted 6 lines

Quoted 2 lines

Apologies, that was indeed a behavioral change introduced in a commit that claimed just to be shuffling code around.

Another complaint about this series: using EINPROGRESS to signal asynchronous locking looks really fishy. How does the filesystem know, that the caller wants to do async locking?

The caller sets a fl_grant callback. But I guess the interesting question is how the caller knows that the filesystem is really going to return the results asynchronously:

Quoted 4 lines

How do we make sure,
that the filesystem (like fuse or 9p, which "blindly" return the error from the server) doesn't return EINPROGRESS even when it's _not_ doing an asynchronous lock?

Right, there's no safeguard there--if fuse returns EINPROGRESS, then we'll wait for a grant callback that's not going to come. It should time out, so that's not a total disaster, but still.

Quoted 8 lines

Anyway, I'm not sure what to do about that.

Quoted 4 lines

I think it would have been much cleaner to have a completely separate interface for async locking, instead of trying to cram that into f_op->lock().

Maybe so. ->lock() had quite a bit crammed into it even before this.

Oh yeah, but at least that was 1:1 with the fcntl interface.

Quoted 6 lines

Quoted 3 lines

Would that be possible to fix now? Or at least make EINPROGRESS a kernel-internal error value (>512), to make it that it has a special meaning for the _kernel only_?

Perhaps so.

Quoted 5 lines

The behavior of lockd will still depend to some degree on the exact error returned from the filesystem--e.g. if you return -EAGAIN from ->lock() without later calling ->fl_grant() it will cause some unexpected delays. (Though again clients will eventually give up and poll for the lock.)

OK, so the semantics of vfs_lock_file() are now:

1) if !FL_SLEEP, then return 0 if granted, -EAGAIN on contention
2) if FL_SLEEP and fl_grant == NULL, then return 0 if granted, block on contention
3) if FL_SLEEP and fl_grant != NULL, then return 0 if granted, on contention:
a) either return -EINPROGRESS and call fl_grant when granted
b) or return -EAGAIN and call fl_notify when the lock needs retrying

As a first step, how about doing the following:

For 3) use LOCK_FILE_ASYNC as a return value. AFAICS there's no reason to distinguish between a) and b). For 1) leave the -EAGAIN error, since in that case no lock was queued.

Here's an untested patch. Comments?

Miklos

---
 fs/gfs2/locking/dlm/plock.c |    2 +-
 fs/lockd/svclock.c          |   13 ++++---------
 fs/locks.c                  |   20 ++++++++++----------
 include/linux/fs.h          |   15 +++++++++++++++
 4 files changed, 30 insertions(+), 20 deletions(-)

Index: linux-2.6/fs/locks.c
===================================================================

fs/locks.c +10 -10

@@ -784,8 +784,10 @@ find_conflict:
 		if (!flock_locks_conflict(request, fl))
 			continue;
 		error = -EAGAIN;
-		if (request->fl_flags & FL_SLEEP)
-			locks_insert_block(fl, request);
+		if (!(request->fl_flags & FL_SLEEP))
+			goto out;
+		error = FILE_LOCK_ASYNC;
+		locks_insert_block(fl, request);
 		goto out;
 	}
 	if (request->fl_flags & FL_ACCESS)

@@ -841,7 +843,7 @@ static int __posix_lock_file(struct inod
 			error = -EDEADLK;
 			if (posix_locks_deadlock(request, fl))
 				goto out;
-			error = -EAGAIN;
+			error = FILE_LOCK_ASYNC;
 			locks_insert_block(fl, request);
 			goto out;
   		}

@@ -1040,7 +1042,7 @@ int posix_lock_file_wait(struct file *fi
 	might_sleep ();
 	for (;;) {
 		error = posix_lock_file(filp, fl, NULL);
-		if ((error != -EAGAIN) || !(fl->fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_ASYNC)
 			break;
 		error = wait_event_interruptible(fl->fl_wait, !fl->fl_next);
 		if (!error)

@@ -1112,9 +1114,7 @@ int locks_mandatory_area(int read_write,
 
 	for (;;) {
 		error = __posix_lock_file(inode, &fl, NULL);
-		if (error != -EAGAIN)
-			break;
-		if (!(fl.fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_ASYNC)
 			break;
 		error = wait_event_interruptible(fl.fl_wait, !fl.fl_next);
 		if (!error) {

@@ -1534,7 +1534,7 @@ int flock_lock_file_wait(struct file *fi
 	might_sleep();
 	for (;;) {
 		error = flock_lock_file(filp, fl);
-		if ((error != -EAGAIN) || !(fl->fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_ASYNC)
 			break;
 		error = wait_event_interruptible(fl->fl_wait, !fl->fl_next);
 		if (!error)

@@ -1806,7 +1806,7 @@ again:
 	else {
 		for (;;) {
 			error = posix_lock_file(filp, file_lock, NULL);
-			if (error != -EAGAIN || cmd == F_SETLK)
+			if (error != FILE_LOCK_ASYNC)
 				break;
 			error = wait_event_interruptible(file_lock->fl_wait,
 					!file_lock->fl_next);

@@ -1934,7 +1934,7 @@ again:
 	else {
 		for (;;) {
 			error = posix_lock_file(filp, file_lock, NULL);
-			if (error != -EAGAIN || cmd == F_SETLK64)
+			if (error != FILE_LOCK_ASYNC)
 				break;
 			error = wait_event_interruptible(file_lock->fl_wait,
 					!file_lock->fl_next);

Index: linux-2.6/fs/gfs2/locking/dlm/plock.c
===================================================================

fs/gfs2/locking/dlm/plock.c +1 -1

@@ -108,7 +108,7 @@ int gdlm_plock(void *lockspace, struct l
 	if (xop->callback == NULL)
 		wait_event(recv_wq, (op->done != 0));
 	else
-		return -EINPROGRESS;
+		return FILE_LOCK_ASYNC;
 
 	spin_lock(&ops_lock);
 	if (!list_empty(&op->list)) {

Index: linux-2.6/fs/lockd/svclock.c
===================================================================

fs/lockd/svclock.c +4 -9

@@ -423,8 +423,8 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru
 			goto out;
 		case -EAGAIN:
 			ret = nlm_lck_denied;
-			break;
-		case -EINPROGRESS:
+			goto out;
+		case FILE_LOCK_ASYNC:
 			if (wait)
 				break;
 			/* Filesystem lock operation is in progress

@@ -439,10 +439,6 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru
 			goto out;
 	}
 
-	ret = nlm_lck_denied;
-	if (!wait)
-		goto out;
-
 	ret = nlm_lck_blocked;
 
 	/* Append to list of blocked */

@@ -520,7 +516,7 @@ nlmsvc_testlock(struct svc_rqst *rqstp, 
 	}
 
 	error = vfs_test_lock(file->f_file, &lock->fl);
-	if (error == -EINPROGRESS) {
+	if (error == FILE_LOCK_ASYNC) {
 		ret = nlmsvc_defer_lock_rqst(rqstp, block);
 		goto out;
 	}

@@ -744,8 +740,7 @@ nlmsvc_grant_blocked(struct nlm_block *b
 	switch (error) {
 	case 0:
 		break;
-	case -EAGAIN:
-	case -EINPROGRESS:
+	case FILE_LOCK_ASYNC:
 		dprintk("lockd: lock still blocked error %d\n", error);
 		nlmsvc_insert_block(block, NLM_NEVER);
 		nlmsvc_release_block(block);

Index: linux-2.6/include/linux/fs.h
===================================================================

include/linux/fs.h +15 -0

@@ -837,6 +837,21 @@ extern spinlock_t files_lock;
 #define FL_SLEEP	128	/* A blocking lock */
 
 /*
+ * FILE_LOCK_ASYNC:
+ *
+ *   Special return value from posix_lock_file() and vfs_lock_file()
+ *   for asynchronous locking.
+ *
+ *   For posix_lock_file() it means, that the lock has been queued on
+ *   the block list.
+ *
+ *   For vfs_lock_file() it means, that the filesystem will call back
+ *   either fl_notify() or fl_granted() when the lock needs to be
+ *   retried or if it has been granted.
+ */
+#define FILE_LOCK_ASYNC 1
+
+/*
  * The POSIX file lock owner is determined by
  * the "struct files_struct" in the thread group
  * (or NULL for no owner - BSD locks).

Quoted 7 lines

Quoted 5 lines

The behavior of lockd will still depend to some degree on the exact error returned from the filesystem--e.g. if you return -EAGAIN from ->lock() without later calling ->fl_grant() it will cause some unexpected delays. (Though again clients will eventually give up and poll for the lock.)

OK, so the semantics of vfs_lock_file() are now:

Not quite; the original idea was that we didn't care about the blocking lock case, since there's already a lock manager callback for that. (It's arguably a minor abuse for us to return NLM_LCK_BLOCKED and then do a grant later even in the case where there's no conflict, but it works.) So we only changed the nonblocking case.

Which may be sacrificing elegance for expediency, and I'm not opposed to fixing that in whatever way seems best. As a start, we could document this better. So:

Quoted 8 lines

1) if !FL_SLEEP, then return 0 if granted, -EAGAIN on contention
2) if FL_SLEEP and fl_grant == NULL, then return 0 if granted, block on contention
3) if FL_SLEEP and fl_grant != NULL, then return 0 if granted, on contention:
a) either return -EINPROGRESS and call fl_grant when granted
b) or return -EAGAIN and call fl_notify when the lock needs retrying

I'd put it this way (after a quick check of the code to convince myself I'm remembering this right...):

1) If FL_SLEEP, then return 0 if granted, and on contention either:
a) block, or
b) return -EAGAIN, and call fl_notify when the lock should be retried.

2) If !FL_SLEEP, then either
a) return a result now, or
b) return -EINPROGRESS, then call fl_grant with the result.

In each case, b) is only of course allowed if the corresponding callback is defined. If you want to support NFS exports, then you should also take option b) when possible. (Though note only 1b is *required* to avoid deadlocks; 2b is basically just a performance enhancement. We could perhaps even ditch 2b if we fixed lockd's BKL reliance and made it multithreaded.)

The names can be confusing, since fl_notify generally results in granting a previously blocked lock, whereas fl_grant just returns the result (positive or negative) of a single lock request. They have the names they do because of the difference in their semantics:

- With fl_grant the filesystem says: I'm giving you the final result of the lock operation. In particular, if I'm telling you it succeeded, that means I've already granted you the lock; don't ask me about it again.

- With fl_notify the filesystem says: something just happened to this lock. I'm not guaranteeing anything, I'm just telling you this would be a good time to try the lock again. Do it and maybe you'll get lucky!

This difference is why the fl_grant() operation takes a status argument (to tell the lock manager what happened), whereas fl_notify() doesn't (since it's just a poke in the ribs, and only the subsequent lock call will give the real status).

Is that explanation helpful?

OK, but I haven't read your patch yet, apologies....

Quoted 30 lines

Quoted 7 lines

Quoted 5 lines

The behavior of lockd will still depend to some degree on the exact error returned from the filesystem--e.g. if you return -EAGAIN from ->lock() without later calling ->fl_grant() it will cause some unexpected delays. (Though again clients will eventually give up and poll for the lock.)

OK, so the semantics of vfs_lock_file() are now:

Not quite; the original idea was that we didn't care about the blocking lock case, since there's already a lock manager callback for that. (It's arguably a minor abuse for us to return NLM_LCK_BLOCKED and then do a grant later even in the case where there's no conflict, but it works.) So we only changed the nonblocking case.

Which may be sacrificing elegance for expediency, and I'm not opposed to fixing that in whatever way seems best. As a start, we could document this better. So:

Quoted 8 lines

1) if !FL_SLEEP, then return 0 if granted, -EAGAIN on contention
2) if FL_SLEEP and fl_grant == NULL, then return 0 if granted, block on contention
3) if FL_SLEEP and fl_grant != NULL, then return 0 if granted, on contention:
a) either return -EINPROGRESS and call fl_grant when granted
b) or return -EAGAIN and call fl_notify when the lock needs retrying

I'd put it this way (after a quick check of the code to convince myself I'm remembering this right...):

Quoted 4 lines

1) If FL_SLEEP, then return 0 if granted, and on contention either:
a) block, or
b) return -EAGAIN, and call fl_notify when the lock should be retried.

Gfs2 seems to return -EINPROGRESS regardless of the FL_SLEEP flag:

Quoted 1 line

OK, but I haven't read your patch yet, apologies....

No problem. Here it is again with some cosmetic fixes and testing.

Miklos
--

Use a special error value FILE_LOCK_DEFERRED to mean that a locking operation returned asynchronously. This is returned by

posix_lock_file() for sleeping locks to mean that the lock has been queued on the block list, and will be woken up when it might become available and needs to be retried (either fl_lmops->fl_notify() is called or fl_wait is woken up).

f_op->lock() to mean either the above, or that the filesystem will call back with fl_lmops->fl_grant() when the result of the locking operation is known. The filesystem can do this for sleeping as well as non-sleeping locks.

This is to make sure, that return values of -EAGAIN and -EINPROGRESS by filesystems are not mistaken to mean an asynchronous locking.

This also makes error handling in fs/locks.c and lockd/svclock.c slightly cleaner.

Signed-off-by Miklos Szeredi <[email protected]>

---
 fs/gfs2/locking/dlm/plock.c |    2 +-
 fs/lockd/svclock.c          |   13 ++++---------
 fs/locks.c                  |   28 ++++++++++++++--------------
 include/linux/fs.h          |    6 ++++++
 4 files changed, 25 insertions(+), 24 deletions(-)

Index: linux-2.6/fs/locks.c
===================================================================

fs/locks.c +14 -14

@@ -784,8 +784,10 @@ find_conflict:
 		if (!flock_locks_conflict(request, fl))
 			continue;
 		error = -EAGAIN;
-		if (request->fl_flags & FL_SLEEP)
-			locks_insert_block(fl, request);
+		if (!(request->fl_flags & FL_SLEEP))
+			goto out;
+		error = FILE_LOCK_DEFERRED;
+		locks_insert_block(fl, request);
 		goto out;
 	}
 	if (request->fl_flags & FL_ACCESS)

@@ -841,7 +843,7 @@ static int __posix_lock_file(struct inod
 			error = -EDEADLK;
 			if (posix_locks_deadlock(request, fl))
 				goto out;
-			error = -EAGAIN;
+			error = FILE_LOCK_DEFERRED;
 			locks_insert_block(fl, request);
 			goto out;
   		}

@@ -1040,7 +1042,7 @@ int posix_lock_file_wait(struct file *fi
 	might_sleep ();
 	for (;;) {
 		error = posix_lock_file(filp, fl, NULL);
-		if ((error != -EAGAIN) || !(fl->fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_DEFERRED)
 			break;
 		error = wait_event_interruptible(fl->fl_wait, !fl->fl_next);
 		if (!error)

@@ -1112,9 +1114,7 @@ int locks_mandatory_area(int read_write,
 
 	for (;;) {
 		error = __posix_lock_file(inode, &fl, NULL);
-		if (error != -EAGAIN)
-			break;
-		if (!(fl.fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_DEFERRED)
 			break;
 		error = wait_event_interruptible(fl.fl_wait, !fl.fl_next);
 		if (!error) {

@@ -1534,7 +1534,7 @@ int flock_lock_file_wait(struct file *fi
 	might_sleep();
 	for (;;) {
 		error = flock_lock_file(filp, fl);
-		if ((error != -EAGAIN) || !(fl->fl_flags & FL_SLEEP))
+		if (error != FILE_LOCK_DEFERRED)
 			break;
 		error = wait_event_interruptible(fl->fl_wait, !fl->fl_next);
 		if (!error)

@@ -1719,17 +1719,17 @@ out:
  * fl_grant is set. Callers expecting ->lock() to return asynchronously
  * will only use F_SETLK, not F_SETLKW; they will set FL_SLEEP if (and only if)
  * the request is for a blocking lock. When ->lock() does return asynchronously,
- * it must return -EINPROGRESS, and call ->fl_grant() when the lock
+ * it must return FILE_LOCK_DEFERRED, and call ->fl_grant() when the lock
  * request completes.
  * If the request is for non-blocking lock the file system should return
- * -EINPROGRESS then try to get the lock and call the callback routine with
- * the result. If the request timed out the callback routine will return a
+ * FILE_LOCK_DEFERRED then try to get the lock and call the callback routine
+ * with the result. If the request timed out the callback routine will return a
  * nonzero return code and the file system should release the lock. The file
  * system is also responsible to keep a corresponding posix lock when it
  * grants a lock so the VFS can find out which locks are locally held and do
  * the correct lock cleanup when required.
  * The underlying filesystem must not drop the kernel lock or call
- * ->fl_grant() before returning to the caller with a -EINPROGRESS
+ * ->fl_grant() before returning to the caller with a FILE_LOCK_DEFERRED
  * return code.
  */
 int vfs_lock_file(struct file *filp, unsigned int cmd, struct file_lock *fl, struct file_lock *conf)

@@ -1806,7 +1806,7 @@ again:
 	else {
 		for (;;) {
 			error = posix_lock_file(filp, file_lock, NULL);
-			if (error != -EAGAIN || cmd == F_SETLK)
+			if (error != FILE_LOCK_DEFERRED)
 				break;
 			error = wait_event_interruptible(file_lock->fl_wait,
 					!file_lock->fl_next);

@@ -1934,7 +1934,7 @@ again:
 	else {
 		for (;;) {
 			error = posix_lock_file(filp, file_lock, NULL);
-			if (error != -EAGAIN || cmd == F_SETLK64)
+			if (error != FILE_LOCK_DEFERRED)
 				break;
 			error = wait_event_interruptible(file_lock->fl_wait,
 					!file_lock->fl_next);

Index: linux-2.6/fs/gfs2/locking/dlm/plock.c
===================================================================

fs/gfs2/locking/dlm/plock.c +1 -1

@@ -108,7 +108,7 @@ int gdlm_plock(void *lockspace, struct l
 	if (xop->callback == NULL)
 		wait_event(recv_wq, (op->done != 0));
 	else
-		return -EINPROGRESS;
+		return FILE_LOCK_DEFERRED;
 
 	spin_lock(&ops_lock);
 	if (!list_empty(&op->list)) {

Index: linux-2.6/fs/lockd/svclock.c
===================================================================

fs/lockd/svclock.c +4 -9

@@ -423,8 +423,8 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru
 			goto out;
 		case -EAGAIN:
 			ret = nlm_lck_denied;
-			break;
-		case -EINPROGRESS:
+			goto out;
+		case FILE_LOCK_DEFERRED:
 			if (wait)
 				break;
 			/* Filesystem lock operation is in progress

@@ -439,10 +439,6 @@ nlmsvc_lock(struct svc_rqst *rqstp, stru
 			goto out;
 	}
 
-	ret = nlm_lck_denied;
-	if (!wait)
-		goto out;
-
 	ret = nlm_lck_blocked;
 
 	/* Append to list of blocked */

@@ -520,7 +516,7 @@ nlmsvc_testlock(struct svc_rqst *rqstp, 
 	}
 
 	error = vfs_test_lock(file->f_file, &lock->fl);
-	if (error == -EINPROGRESS) {
+	if (error == FILE_LOCK_DEFERRED) {
 		ret = nlmsvc_defer_lock_rqst(rqstp, block);
 		goto out;
 	}

@@ -744,8 +740,7 @@ nlmsvc_grant_blocked(struct nlm_block *b
 	switch (error) {
 	case 0:
 		break;
-	case -EAGAIN:
-	case -EINPROGRESS:
+	case FILE_LOCK_DEFERRED:
 		dprintk("lockd: lock still blocked error %d\n", error);
 		nlmsvc_insert_block(block, NLM_NEVER);
 		nlmsvc_release_block(block);

Index: linux-2.6/include/linux/fs.h
===================================================================

include/linux/fs.h +6 -0

@@ -837,6 +837,12 @@ extern spinlock_t files_lock;
 #define FL_SLEEP	128	/* A blocking lock */
 
 /*
+ * Special return value from posix_lock_file() and vfs_lock_file() for
+ * asynchronous locking.
+ */
+#define FILE_LOCK_DEFERRED 1
+
+/*
  * The POSIX file lock owner is determined by
  * the "struct files_struct" in the thread group
  * (or NULL for no owner - BSD locks).

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

1) if !FL_SLEEP, then return 0 if granted, -EAGAIN on contention
2) if FL_SLEEP and fl_grant == NULL, then return 0 if granted, block on contention
3) if FL_SLEEP and fl_grant != NULL, then return 0 if granted, on contention:
a) either return -EINPROGRESS and call fl_grant when granted
b) or return -EAGAIN and call fl_notify when the lock needs retrying

I'd put it this way (after a quick check of the code to convince myself I'm remembering this right...):

Quoted 6 lines

Quoted 4 lines

1) If FL_SLEEP, then return 0 if granted, and on contention either:
a) block, or
b) return -EAGAIN, and call fl_notify when the lock should be retried.

Gfs2 seems to return -EINPROGRESS regardless of the FL_SLEEP flag:

Oops, you're right; in FL_SLEEP case fs/lockd/svclock.c:nlmsvc_lock() returns NLM_LCK_BLOCKED. I believe it'll get an fl_grant() callback after that and do a grant call back to the client, but I haven't checked....

Note, as has been pointed out by Mark Snitzer
(http://article.gmane.org/gmane.linux.nfs/17801/), this limits the kind of error reporting the filesystem can do--if it needs to block on the initial lock call, it has to commit to queueing up, and eventually granting, the lock.

Quoted 3 lines

Quoted 1 line

OK, but I haven't read your patch yet, apologies....

No problem. Here it is again with some cosmetic fixes and testing.

Thanks! Ping me in a couple days if I haven't made any comments. From a quick skim the GFS2 change and the error return change both seem reasonable.

I need to a real GFS2 testing setup.... (Did you test GFS2 locking specifically?)