Skip to contentSkip to conversation

exportfs: fix handling of rename race in reconnect_one()

PATCH In review Amir Goldstein 4 participants 10 messages Jan 26, 2020 – Mar 13, 2020

Subsystems (2)
lore
Share & tools
lore

Amir Goldstein

author reviewer
#0 lore raw

PATCH 1/1 exportfs: fix handling of rename race in reconnect_one()

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

Cc Christoph Hellwig <[email protected]>
Cc J. Bruce Fields <[email protected]>
Fixes 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning")
---
 fs/exportfs/expfs.c | 2 ++
 1 file changed, 2 insertions(+)
fs/exportfs/expfs.c +2 -0
index 2dd55b172d57..25a09bacf9c1 100644
@@ -149,6 +149,8 @@ static struct dentry *reconnect_one(struct vfsmount *mnt,
 	if (IS_ERR(tmp)) {
 		dprintk("%s: lookup failed: %d\n", __func__, PTR_ERR(tmp));
 		err = PTR_ERR(tmp);
+		if (err == -ENOENT)
+			goto out_reconnected;
 		goto out_err;
 	}
 	if (tmp != dentry) {
Signature · 1 line
2.17.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 5 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

Quoted 8 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

Looks good:

#2 lore raw
In reply to Amir Goldstein

Thanks for spotting this!

Quoted 4 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

A comment that -ENOENT means the parent's gone might be helpful.

But are we sure -ENOENT is what every filesystem returns in the case the parent was deleted? And are we sure there aren't other cases that should be handled similarly to -ENOENT?

Quoted 4 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

I wonder whether it might be safest to take the out_reconnected case on any error, not just -ENOENT.

Looking further back through the history.... Looks like the missing PTR_ERR(tmp) was just a mistake, introduced in 2013 by my bbf7a8a3562f "exportfs: move most of reconnect_path to helper function". So the historical behavior was always to bail on error.

The old code still did a DCACHE_DISCONNECTED check on the target dentry in that case and returned success if it found that already cleared, but we can't necessarily rely on DCACHE_DISCONNECTED being cleared immediately, so the old code was probably still vulnerable to the race you saw.

There's not much value in preserving the error as exportfs_decode_fh() ends up turning everything into ENOMEM or ESTALE for some reason.

Hm.--b.

Quoted 28 lines

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

Cc Christoph Hellwig <[email protected]>
Cc J. Bruce Fields <[email protected]>
Fixes 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning")
---
 fs/exportfs/expfs.c | 2 ++
 1 file changed, 2 insertions(+)
fs/exportfs/expfs.c +2 -0
index 2dd55b172d57..25a09bacf9c1 100644
@@ -149,6 +149,8 @@ static struct dentry *reconnect_one(struct vfsmount *mnt,
 	if (IS_ERR(tmp)) {
 		dprintk("%s: lookup failed: %d\n", __func__, PTR_ERR(tmp));
 		err = PTR_ERR(tmp);
+		if (err == -ENOENT)
+			goto out_reconnected;
 		goto out_err;
 	}
 	if (tmp != dentry) {
Signature · 1 line
2.17.1
author reviewer
#3 lore raw
Quoted 10 lines · J. Bruce Fields

Thanks for spotting this!

Quoted 4 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

A comment that -ENOENT means the parent's gone might be helpful.

It doesn't have to mean that, but that's the most obvious case.

Quoted 3 lines

But are we sure -ENOENT is what every filesystem returns in the case the parent was deleted?

No, it's what __lookup_slow() returns if parent is dead. Most filesystems do not return -ENOENT for lookup, but a negative dentry on NULL. I am not sure which filesystems return -ENOENT. A short survey of NFS exporting fs I didn't find any.

Quoted 3 lines

And are we sure there aren't other cases that
should be handled similarly to -ENOENT?

Not sure, but ENOENT is the most obvious one for rename race.

Quoted 8 lines
Quoted 4 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

I wonder whether it might be safest to take the out_reconnected case on any error, not just -ENOENT.

I wondered that as well, but preferred to follow the precedent.

Quoted 11 lines

Looking further back through the history.... Looks like the missing PTR_ERR(tmp) was just a mistake, introduced in 2013 by my bbf7a8a3562f "exportfs: move most of reconnect_path to helper function". So the historical behavior was always to bail on error.

The old code still did a DCACHE_DISCONNECTED check on the target dentry in that case and returned success if it found that already cleared, but we can't necessarily rely on DCACHE_DISCONNECTED being cleared immediately, so the old code was probably still vulnerable to the race you saw.

Yeh, I started to try and document history, but since there seemed to be no point where behavior looked sane I gave up.

Quoted 3 lines

There's not much value in preserving the error as exportfs_decode_fh() ends up turning everything into ENOMEM or ESTALE for some reason.

You signed up on this reason...

Thanks,
Amir.

commit 09bb8bfffd29c3dffb72bc2c69a062dfb1ae624c

Author: NeilBrown <[email protected]>
Date:   Thu Aug 4 10:19:06 2016 +1000

exportfs: be careful to only return expected errors.

When nfsd calls fh_to_dentry, it expect ESTALE or ENOMEM as errors. In particular it can be tempting to return ENOENT, but this is not handled well by nfsd.

Rather than requiring strict adherence to error code code filesystems, treat all unexpected error codes the same as ESTALE. This is safest.

#4 lore raw
Quoted 10 lines · Amir Goldstein
Quoted 8 lines
Quoted 4 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

I wonder whether it might be safest to take the out_reconnected case on any error, not just -ENOENT.

I wondered that as well, but preferred to follow the precedent.

I can live with that.

Quoted 5 lines
Quoted 3 lines

There's not much value in preserving the error as exportfs_decode_fh() ends up turning everything into ENOMEM or ESTALE for some reason.

You signed up on this reason...

Hah, I forgot that one.

--b.

Quoted 20 lines

Thanks,
Amir.

commit 09bb8bfffd29c3dffb72bc2c69a062dfb1ae624c

Author: NeilBrown <[email protected]>
Date:   Thu Aug 4 10:19:06 2016 +1000

exportfs: be careful to only return expected errors.

When nfsd calls fh_to_dentry, it expect ESTALE or ENOMEM as errors. In particular it can be tempting to return ENOENT, but this is not handled well by nfsd.

Rather than requiring strict adherence to error code code filesystems, treat all unexpected error codes the same as ESTALE. This is safest.

author reviewer
#5 lore raw
Quoted 7 lines · J. Bruce Fields
Quoted 5 lines · Amir Goldstein
Quoted 5 lines
Quoted 4 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I wonder whether it might be safest to take the out_reconnected case on any error, not just -ENOENT.

I wondered that as well, but preferred to follow the precedent.

I can live with that.

Will you take this patch through your tree,
or do you want me to re-post to Al?
With your Reviewed-by?

Thanks,
Amir.

#6 lore raw
Quoted 12 lines · Amir Goldstein
Quoted 11 lines · J. Bruce Fields
Quoted 9 lines · Amir Goldstein
Quoted 8 lines
Quoted 4 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

I wonder whether it might be safest to take the out_reconnected case on any error, not just -ENOENT.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I wondered that as well, but preferred to follow the precedent.

I can live with that.

Will you take this patch through your tree,
or do you want me to re-post to Al?

If Al wants to delegate exportfs/ patches to nfsd maintainers that's OK by me, but in the past I think it's always him that's taken them.

Quoted 1 line

With your Reviewed-by?

That'd be fine. Thanks!

--b.

maintainer
#7 lore raw
In reply to Amir Goldstein
Quoted 5 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

Quoted 8 lines

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

It's not that simple, unfortunately. For one thing, lookup_one_len_unlocked() will normally return a negative dentry, not ERR_PTR(-ENOENT). For another, it *can* fail for any number of other reasons (-ENOMEM, for example), without anyone having ever looked it up.

So I agree that the damn thing needs work, but I don't believe that this is the right fix.

author reviewer
#8 lore raw
Quoted 12 lines · Al Viro
Quoted 10 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Quoted 6 lines
Quoted 3 lines

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

It's not that simple, unfortunately. For one thing, lookup_one_len_unlocked() will normally return a negative dentry, not ERR_PTR(-ENOENT).

Which is why this fix is mostly relevant to removed directories. negative dentry case should be handled correctly by bellow:

        if (tmp != dentry) {
Quoted 3 lines

For another,
it *can* fail for any number of other reasons (-ENOMEM, for example), without anyone having ever looked it up.

Yes, but why should we care to NOT return an error in case of ENOMEM. The question is are there other errors that we can say "we can let this slide" as long as the dentry is connected?

I certainly don't mind going to out_reconnected for any error and that includes the error from exportfs_get_name(). My patch checks only the rename race case because this is what this function has done so far and this is what the big comment in out_reconnect is about.

Thanks,
Amir.

author reviewer
#9 lore raw
Quoted 14 lines · Amir Goldstein
Quoted 12 lines · Al Viro
Quoted 10 lines · Amir Goldstein

If a disconnected dentry gets looked up and renamed between the call to exportfs_get_name() and lookup_one_len_unlocked(), and if also lookup_one_len_unlocked() returns ERR_PTR(-ENOENT), maybe because old parent was deleted, we return an error, although dentry may be connected.

Commit 909e22e05353 ("exportfs: fix 'passing zero to ERR_PTR()' warning") changes this behavior from always returning success, regardless if dentry was reconnected by somoe other task, to always returning a failure.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Change the lookup error handling to match that of exportfs_get_name() error handling and return success after getting -ENOENT and verifying that some other task has connected the dentry for us.

It's not that simple,

Al,

Ping.
Are you sure it is not that simple for all practical cases? Please take a closer look.

My change attempts to handle a real rename race similar to how it was handled before the "Fixes" commit.
This is Acked by Bruce and Christoph.

Please see my arguments below.

Thanks,
Amir.

Quoted 24 lines

unfortunately. For one thing, lookup_one_len_unlocked()

Quoted 1 line

will normally return a negative dentry, not ERR_PTR(-ENOENT).

Which is why this fix is mostly relevant to removed directories. negative dentry case should be handled correctly by bellow:

        if (tmp != dentry) {
Quoted 3 lines

For another,
it *can* fail for any number of other reasons (-ENOMEM, for example), without anyone having ever looked it up.

Yes, but why should we care to NOT return an error in case of ENOMEM. The question is are there other errors that we can say "we can let this slide" as long as the dentry is connected?

I certainly don't mind going to out_reconnected for any error and that includes the error from exportfs_get_name(). My patch checks only the rename race case because this is what this function has done so far and this is what the big comment in out_reconnect is about.

Thanks,
Amir.