Skip to contentSkip to conversation

3.7-rc1 NFSv3/sec=krb5 mkdir failure

DISCUSSION J. Bruce Fields 3 participants 13 messages Oct 16, 2012 – Nov 16, 2012

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

J. Bruce Fields

author
#0 lore raw

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

--b.

author
#1 lore raw

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 17 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

--b.

#2 lore raw
Quoted 7 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 20 lines

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 15 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Cheers
Trond

author
#3 lore raw
Quoted 22 lines · Trond
Quoted 22 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 10 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
		server# ls -l /exports/ext4|grep TMP
		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
		server#

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

--b.

#4 lore raw
Quoted 34 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 26 lines · Trond
Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 14 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

Cheers
Trond

author
#5 lore raw
Quoted 34 lines · Trond
Quoted 34 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 26 lines · Trond
Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 14 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4 /mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

ACLs on the export and the new directory are both trivial:

[root@pip1 ~]# ls -ld /exports/ext4/ drwxrwxrwx. 8 root root 4096 Oct 24 16:12 /exports/ext4/ [root@pip1 ~]# getfacl /exports/ext4
getfacl: Removing leading '/' from absolute path names # file: exports/ext4
# owner: root
# group: root
user::rwx
group::rwx
other::rwx

[root@pip1 ~]# ls -ld /exports/ext4/TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 24 16:12 /exports/ext4/TMP
[root@pip1 ~]# getfacl /exports/ext4/TMP
getfacl: Removing leading '/' from absolute path names # file: exports/ext4/TMP
# owner: nfsnobody
# group: nfsnobody
user::rwx
group::r-x
other::r-x

[root@pip1 ~]#

--b.

#6 lore raw
In reply to J. Bruce Fields
Quoted 44 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 37 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 29 lines · Trond
Quoted 28 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 16 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

Cheers
Trond

author
#7 lore raw
Quoted 46 lines · Trond
Quoted 46 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 39 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 30 lines · Trond
Quoted 7 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 21 lines

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 16 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

That's on the client mount command? No difference.

--b.

author
#8 lore raw
Quoted 50 lines · J. Bruce Fields
Quoted 49 lines · Trond
Quoted 49 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 41 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 23 lines · Trond
Quoted 23 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 11 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
		server# ls -l /exports/ext4|grep TMP
		drwxr-xr-x  2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP
		server#

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

That's on the client mount command? No difference.

By the way, I managed to do a little bisecting while working on something else today, and blame landed on Chuck's ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce rpc_clone_client_set_auth()". Which makes some sense if it's an ACL problem, and indeed testing on that commit finds success with -noacl, failure without.

I'm not sure if that explains the failure I was seeing on 3.7-rc1, since there I didn't see any ACL traffic, and still got a failure. (And -noacl didn't help.)

--b.

#9 lore raw
Quoted 57 lines · J. Bruce Fields
Quoted 56 lines · J. Bruce Fields
Quoted 54 lines · Trond
Quoted 44 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 37 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 29 lines · Trond
Quoted 28 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 16 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

Quoted 8 lines
Quoted 1 line

That's on the client mount command? No difference.

By the way, I managed to do a little bisecting while working on something else today, and blame landed on Chuck's ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce rpc_clone_client_set_auth()". Which makes some sense if it's an ACL problem, and indeed testing on that commit finds success with -noacl, failure without.

After two weeks, Bruce and I were finally able to catch up in person.

I've reproduced this on 3.7-rc5 using cthon basic tests. The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists. That's in line with the kind of change that's in commit ba9b584c1.

Quoted 3 lines

I'm not sure if that explains the failure I was seeing on 3.7-rc1, since there I didn't see any ACL traffic, and still got a failure. (And -noacl didn't help.)

The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case. The failure prevents any ACL request from succeeding.

Anyway, I'll have a deeper look at this tomorrow.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
author
#10 lore raw
Quoted 73 lines · Chuck Lever
Quoted 57 lines · J. Bruce Fields
Quoted 56 lines · J. Bruce Fields
Quoted 54 lines · Trond
Quoted 44 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 37 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 29 lines · Trond
Quoted 28 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 16 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP server#

# mkdir /mnt/TMP
mkdir: cannot create directory `/mnt/TMP': Permission denied

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

Quoted 8 lines
Quoted 1 line

That's on the client mount command? No difference.

By the way, I managed to do a little bisecting while working on something else today, and blame landed on Chuck's ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce rpc_clone_client_set_auth()". Which makes some sense if it's an ACL problem, and indeed testing on that commit finds success with -noacl, failure without.

After two weeks, Bruce and I were finally able to catch up in person.

I've reproduced this on 3.7-rc5 using cthon basic tests. The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists. That's in line with the kind of change that's in commit ba9b584c1.

Quoted 7 lines
Quoted 3 lines

I'm not sure if that explains the failure I was seeing on 3.7-rc1, since there I didn't see any ACL traffic, and still got a failure. (And -noacl didn't help.)

The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case. The failure prevents any ACL request from succeeding.

Anyway, I'll have a deeper look at this tomorrow.

Great, thanks for taking another look.

--b.

#11 lore raw
In reply to Chuck Lever

T24gVGh1LCAyMDEyLTExLTE1IGF0IDIxOjAzIC0wNTAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g T24gT2N0IDI4LCAyMDEyLCBhdCAxMjoxNSBQTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRzQGZp ZWxkc2VzLm9yZz4gd3JvdGU6DQo+IA0KPiA+IE9uIFdlZCwgT2N0IDI0LCAyMDEyIGF0IDA0OjQw OjU5UE0gLTA0MDAsIEouIEJydWNlIEZpZWxkcyB3cm90ZToNCj4gPj4gT24gV2VkLCBPY3QgMjQs IDIwMTIgYXQgMDg6MzQ6MzdQTSArMDAwMCwgTXlrbGVidXN0LCBUcm9uZCB3cm90ZToNCj4gPj4+ PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+IEZyb206IE15a2xlYnVzdCwgVHJv bmQNCj4gPj4+PiBTZW50OiBXZWRuZXNkYXksIE9jdG9iZXIgMjQsIDIwMTIgNDozMSBQTQ0KPiA+ Pj4+IFRvOiAnSi4gQnJ1Y2UgRmllbGRzJw0KPiA+Pj4+IENjOiBsaW51eC1uZnNAdmdlci5rZXJu ZWwub3JnOyBTY2h1bWFrZXIsIEJyeWFuDQo+ID4+Pj4gU3ViamVjdDogUkU6IDMuNy1yYzEgTkZT djMvc2VjPWtyYjUgbWtkaXIgZmFpbHVyZQ0KPiA+Pj4+IA0KPiA+Pj4+PiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+PiBGcm9tOiBKLiBCcnVjZSBGaWVsZHMgW21haWx0bzpiZmll bGRzQGZpZWxkc2VzLm9yZ10NCj4gPj4+Pj4gU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDI0LCAy MDEyIDQ6MTUgUE0NCj4gPj4+Pj4gVG86IE15a2xlYnVzdCwgVHJvbmQNCj4gPj4+Pj4gQ2M6IGxp bnV4LW5mc0B2Z2VyLmtlcm5lbC5vcmc7IFNjaHVtYWtlciwgQnJ5YW4NCj4gPj4+Pj4gU3ViamVj dDogUmU6IDMuNy1yYzEgTkZTdjMvc2VjPWtyYjUgbWtkaXIgZmFpbHVyZQ0KPiA+Pj4+PiANCj4g Pj4+Pj4gT24gV2VkLCBPY3QgMjQsIDIwMTIgYXQgMDg6MDc6NTVQTSArMDAwMCwgTXlrbGVidXN0 LCBUcm9uZCB3cm90ZToNCj4gPj4+Pj4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+ Pj4+Pj4+IEZyb206IGxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5lbC5vcmcgW21haWx0bzpsaW51 eC1uZnMtDQo+ID4+Pj4+Pj4gb3duZXJAdmdlci5rZXJuZWwub3JnXSBPbiBCZWhhbGYgT2YgSi4g QnJ1Y2UgRmllbGRzDQo+ID4+Pj4+Pj4gU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDI0LCAyMDEy IDQ6MDMgUE0NCj4gPj4+Pj4+PiBUbzogbGludXgtbmZzQHZnZXIua2VybmVsLm9yZzsgTXlrbGVi dXN0LCBUcm9uZDsgU2NodW1ha2VyLCBCcnlhbg0KPiA+Pj4+Pj4+IFN1YmplY3Q6IFJlOiAzLjct cmMxIE5GU3YzL3NlYz1rcmI1IG1rZGlyIGZhaWx1cmUNCj4gPj4+Pj4+PiANCj4gPj4+Pj4+PiBB bnlvbmUgZ2V0IGEgY2hhbmNlIHRvIGxvb2sgYXQgdGhpcz8gIEl0IHNlZW1zIHZlcnkgcmVwcm9k dWNlYWJsZS4NCj4gPj4+Pj4+PiANCj4gPj4+Pj4+PiAtLWIuDQo+ID4+Pj4+Pj4gDQo+ID4+Pj4+ Pj4gT24gVHVlLCBPY3QgMTYsIDIwMTIgYXQgMDg6NTg6MzJBTSAtMDQwMCwgYmZpZWxkcyB3cm90 ZToNCj4gPj4+Pj4+Pj4gT24gMy43LXJjMToNCj4gPj4+Pj4+Pj4gDQo+ID4+Pj4+Pj4+IAljbGll bnQjIG1vdW50IC10bmZzIC1vc2VjPWtyYjUsdmVycz0zIHNlcnZlcjovZXhwb3J0cy9leHQ0DQo+ ID4+Pj4+Pj4+IC9tbnQvDQo+ID4+Pj4+Pj4+IA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMgbHMgLWwg L2V4cG9ydHMvZXh0NHxncmVwIFRNUA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMNCj4gPj4+Pj4+Pj4g DQo+ID4+Pj4+Pj4+IAkjIG1rZGlyIC9tbnQvVE1QDQo+ID4+Pj4+Pj4+IAlta2RpcjogY2Fubm90 IGNyZWF0ZSBkaXJlY3RvcnkgYC9tbnQvVE1QJzogUGVybWlzc2lvbiBkZW5pZWQNCj4gPj4+Pj4+ Pj4gDQo+ID4+Pj4+Pj4+IAkJc2VydmVyIyBscyAtbCAvZXhwb3J0cy9leHQ0fGdyZXAgVE1QDQo+ ID4+Pj4+Pj4+IAkJZHJ3eHIteHIteCAgMiBuZnNub2JvZHkgbmZzbm9ib2R5IDQwOTYgT2N0IDE2 IDA4OjU2IFRNUA0KPiA+Pj4+Pj4+PiAJCXNlcnZlciMNCj4gPj4+Pj4+Pj4gDQo+ID4+Pj4+Pj4+ IFdpcmVzaGFyayBhbHNvIHNob3dzIHRoYXQgdGhlIGNyZWF0ZSBzdWNjZWVkcy4NCj4gPj4+Pj4+ IA0KPiA+Pj4+Pj4gQ2FuIHlvdSBzaGFyZSB0aGUgd2lyZXNoYXJrIHRyYWNlPw0KPiA+Pj4+PiAN Cj4gPj4+Pj4gU3VyZS4gIFRoaXMgY292ZXJzIHRoZSBtb3VudCBhbmQgbWtkaXIuICBUaGUgbWtk aXIgY2FsbCBhbmQgcmVwbHkgYXJlDQo+ID4+Pj4+IGluIGZyYW1lcyA3NyBhbmQgNzguDQo+ID4+ Pj4gDQo+ID4+Pj4gSG1tLi4uLiBDYW4geW91IHBsZWFzZSBjaGVjayBpZiB0aGUgQUNMIGlzIGJl aW5nIHNldCBjb3JyZWN0bHkgb24gdGhlIHNlcnZlcj8gSQ0KPiA+Pj4+IHN1c3BlY3QgdGhhdCBt aWdodCBiZSB0aGUgc291cmNlIG9mIHRoZSBlcnJvci4NCj4gPj4+PiANCj4gPj4+IA0KPiA+Pj4g SW4gZmFjdCwgY2FuIHlvdSBzZWUgaWYgbW91bnRpbmcgd2l0aCAnLW9ub2FjbCcgY2F1c2VzIHRo ZSB3aG9sZSB0aGluZyB0byBzdWNjZWVkPw0KPiA+PiANCj4gPj4gVGhhdCdzIG9uIHRoZSBjbGll bnQgbW91bnQgY29tbWFuZD8gIE5vIGRpZmZlcmVuY2UuDQo+ID4gDQo+ID4gQnkgdGhlIHdheSwg SSBtYW5hZ2VkIHRvIGRvIGEgbGl0dGxlIGJpc2VjdGluZyB3aGlsZSB3b3JraW5nIG9uDQo+ID4g c29tZXRoaW5nIGVsc2UgdG9kYXksIGFuZCBibGFtZSBsYW5kZWQgb24gQ2h1Y2sncw0KPiA+IGJh OWI1ODRjMWRjMzc4NTFkOWM2Y2E2ZDBkMmNjYmE1NWQ5YWFkMDQgIlNVTlJQQzogSW50cm9kdWNl DQo+ID4gcnBjX2Nsb25lX2NsaWVudF9zZXRfYXV0aCgpIi4gIFdoaWNoIG1ha2VzIHNvbWUgc2Vu c2UgaWYgaXQncyBhbiBBQ0wNCj4gPiBwcm9ibGVtLCBhbmQgaW5kZWVkIHRlc3Rpbmcgb24gdGhh dCBjb21taXQgZmluZHMgc3VjY2VzcyB3aXRoIC1ub2FjbCwNCj4gPiBmYWlsdXJlIHdpdGhvdXQu DQo+IA0KPiBBZnRlciB0d28gd2Vla3MsIEJydWNlIGFuZCBJIHdlcmUgZmluYWxseSBhYmxlIHRv IGNhdGNoIHVwIGluIHBlcnNvbi4NCj4gDQo+IEkndmUgcmVwcm9kdWNlZCB0aGlzIG9uIDMuNy1y YzUgdXNpbmcgY3Rob24gYmFzaWMgdGVzdHMuICBUaGUgZmlyc3QgZ2V0YWNsIG9wZXJhdGlvbiBm YWlscyBiZWNhdXNlIGl0J3MgbWlzdGFrZW5seSBhdHRlbXB0aW5nIHRvIHNldCB1cCBhIGZyZXNo IEdTUyBjb250ZXh0IG9uIGEgdHJhbnNwb3J0IHdoZXJlIG9uZSBhbHJlYWR5IGV4aXN0cy4gIFRo YXQncyBpbiBsaW5lIHdpdGggdGhlIGtpbmQgb2YgY2hhbmdlIHRoYXQncyBpbiBjb21taXQgYmE5 YjU4NGMxLg0KDQpXaHkgc2hvdWxkbid0IHdlIGJlIGFibGUgdG8gY29wZSB3aXRoIG11bHRpcGxl IEdTUyBzZXNzaW9ucyBvbiB0aGUgc2FtZQ0KdHJhbnNwb3J0Pw0KDQo+ID4gSSdtIG5vdCBzdXJl IGlmIHRoYXQgZXhwbGFpbnMgdGhlIGZhaWx1cmUgSSB3YXMgc2VlaW5nIG9uIDMuNy1yYzEsIHNp bmNlDQo+ID4gdGhlcmUgSSBkaWRuJ3Qgc2VlIGFueSBBQ0wgdHJhZmZpYywgYW5kIHN0aWxsIGdv dCBhIGZhaWx1cmUuICAoQW5kDQo+ID4gLW5vYWNsIGRpZG4ndCBoZWxwLikNCj4gDQo+IFRoZSBm YWlsdXJlIG9jY3VycyBvbiB0aGUgY2xpZW50IGp1c3QgYmVmb3JlIHRoZSBnZXRhY2wgcmVxdWVz dCBpcyBpc3N1ZWQsIHNvIHlvdSB3b24ndCBzZWUgYW55IEFDTC1yZWxhdGVkIG5ldHdvcmsgdHJh ZmZpYyBpbiB0aGUgZmFpbHVyZSBjYXNlLiAgVGhlIGZhaWx1cmUgcHJldmVudHMgYW55IEFDTCBy ZXF1ZXN0IGZyb20gc3VjY2VlZGluZy4NCg0KSXMgaXQgZ3NzZCB0aGF0IGlzIGZhaWxpbmcgdGhl bj8NCg0KPiBBbnl3YXksIEknbGwgaGF2ZSBhIGRlZXBlciBsb29rIGF0IHRoaXMgdG9tb3Jyb3cu DQoNClRoYW5rcyENCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFp bnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBw LmNvbQ0K

#12 lore raw

Sent from my iPad

Quoted 69 lines · Trond
Quoted 68 lines · Chuck Lever
Quoted 65 lines · J. Bruce Fields
Quoted 50 lines · J. Bruce Fields
Quoted 49 lines · Trond
Quoted 49 lines

-----Original Message-----
From: Myklebust, Trond
Sent: Wednesday, October 24, 2012 4:31 PM
To: 'J. Bruce Fields'
Cc: [email protected]; Schumaker, Bryan
Subject: RE: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 41 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, October 24, 2012 4:15 PM
To: Myklebust, Trond
Cc: [email protected]; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Quoted 23 lines · Trond
Quoted 23 lines

-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of J. Bruce Fields
Sent: Wednesday, October 24, 2012 4:03 PM
To: [email protected]; Myklebust, Trond; Schumaker, Bryan
Subject: Re: 3.7-rc1 NFSv3/sec=krb5 mkdir failure

Anyone get a chance to look at this? It seems very reproduceable.

--b.

Quoted 11 lines · bfields

On 3.7-rc1:

client# mount -tnfs -osec=krb5,vers=3 server:/exports/ext4
/mnt/

server# ls -l /exports/ext4|grep TMP
server#

   # mkdir /mnt/TMP
   mkdir: cannot create directory `/mnt/TMP': Permission denied
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

server# ls -l /exports/ext4|grep TMP drwxr-xr-x 2 nfsnobody nfsnobody 4096 Oct 16 08:56 TMP server#

Wireshark also shows that the create succeeds.

Can you share the wireshark trace?

Sure. This covers the mount and mkdir. The mkdir call and reply are in frames 77 and 78.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Hmm.... Can you please check if the ACL is being set correctly on the server? I suspect that might be the source of the error.

In fact, can you see if mounting with '-onoacl' causes the whole thing to succeed?

That's on the client mount command? No difference.

By the way, I managed to do a little bisecting while working on something else today, and blame landed on Chuck's ba9b584c1dc37851d9c6ca6d0d2ccba55d9aad04 "SUNRPC: Introduce rpc_clone_client_set_auth()". Which makes some sense if it's an ACL problem, and indeed testing on that commit finds success with -noacl, failure without.

Quoted 6 lines
Quoted 3 lines

After two weeks, Bruce and I were finally able to catch up in person.

I've reproduced this on 3.7-rc5 using cthon basic tests. The first getacl operation fails because it's mistakenly attempting to set up a fresh GSS context on a transport where one already exists. That's in line with the kind of change that's in commit ba9b584c1.

Why shouldn't we be able to cope with multiple GSS sessions on the same transport?

Perhaps we should be able to, in general. But the successful case here does not attempt to create a new context, it simply uses one that is already associated with the transport. That indicates that the kernel is making an incorrect upcall request perhaps because the new code is not cloning the RPC client correctly.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

I'm not sure if that explains the failure I was seeing on 3.7-rc1, since there I didn't see any ACL traffic, and still got a failure. (And -noacl didn't help.)

The failure occurs on the client just before the getacl request is issued, so you won't see any ACL-related network traffic in the failure case. The failure prevents any ACL request from succeeding.

Is it gssd that is failing then?

The upcall fails, yes. That is translated into an immediate failure of the getacl operation.