sec=krb5 mount problem with nfs-utils 1.2.3 on client side
Subsystems (2)
Hi,
I recently updated my nfs clients and server to the new nfs-utls Version 1.2.3 and then also tested a sec=krb5 mount via nfs4 again. For some reason the mount failed and I got the following message on the server.
rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/[email protected]
DEBUG: serialize_krb5_ctx: lucid version! prepare_krb5_rfc4121_buffer: protocol 1 prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32 doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1302858643 (35979 from now), clnt: [email protected], uid: -1, gid: -1, num aux grps: 0:
: qword_eol: fflush failed: errno 22 (Invalid argument)
WARNING: error writing to downcall channel /proc/net/rpc/auth.rpcsec.context/channel: Invalid argument sending null reply
finished handling null request
<repeated 4 times>
Since it worked before I downgraded the versions on both the client and the server to see when it started working again.
Server with 1.2.3 and client with 1.2.2 did the trick
rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/[email protected]
DEBUG: serialize_krb5_ctx: lucid version! prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
doing downcall
mech: krb5, hndl len: 4, ctx len 85, timeout: 1302858893 (36000 from now), clnt: [email protected], uid: -1, gid: -1, num aux grps: 0:
sending null reply
finished handling null request
entering poll
So apparently with 1.2.2 on the client side a different enctype is select. I searched in the archives and saw that this problem was already known but I did not see any fixes. Both client and server have aes_generic enabled and loaded I also made sure that the correct keys on the kerberos side are available.
Valid starting Expires Service principal 04/15/11
01:14:53 04/15/11 11:14:53 krbtgt/[email protected] renew until 04/16/11 01:14:53, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1 04/15/11 01:14:53 04/15/11 11:14:53 nfs/[email protected] renew until 04/16/11 01:14:53, Etype (skey, tkt): des-cbc-crc, aes256-cts-hmac-sha1-96
So right now I am little bit stumped
* why another enctype is used
* why it does not succeed with the new enctype
Both machines are running kernel 2.6.38 on debian sid with the latest Kerberos 1.9
If you need more information please tell me.
Kind regards,
Michael Guntsche
Michael,
I've posted an email with the same question here a few weeks ago, but I haven't been able to investigate what the problem is. The only difference in my setup, that servers have kernel 2.6.36 the rest is exactly the same and exactly the same error. See thread with the subject "rpc.svcgssd problem after updating client 1.2.2->1.2.3".
Regards,
Vladimir.
Quoted 77 lines · Michael Guntsche
Hi,
I recently updated my nfs clients and server to the new nfs-utls Version 1.2.3 and then also tested a sec=krb5 mount via nfs4 again. For some reason the mount failed and I got the following message on the server.
rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/[email protected]
DEBUG:serialize_krb5_ctx: lucid version!prepare_krb5_rfc4121_buffer: protocol 1prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32 doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1302858643 (35979 from now), clnt: [email protected], uid: -1, gid: -1, num aux grps: 0:
:qword_eol: fflush failed: errno 22 (Invalid argument)
WARNING: error writing to downcall channel/proc/net/rpc/auth.rpcsec.context/channel: Invalid argument sending null reply
finished handling null request
<repeated 4 times>Since it worked before I downgraded the versions on both the client and the server to see when it started working again.
Server with 1.2.3 and client with 1.2.2 did the trick
rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/[email protected]
DEBUG:serialize_krb5_ctx: lucid version!prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
doing downcall
mech: krb5, hndl len: 4, ctx len 85, timeout: 1302858893 (36000 from now), clnt: [email protected], uid: -1, gid: -1, num aux grps: 0:
sending null reply
finished handling null request
entering pollSo apparently with 1.2.2 on the client side a different enctype is select. I searched in the archives and saw that this problem was already known but I did not see any fixes. Both client and server have
aes_genericenabled and loaded I also made sure that the correct keys on the kerberos side are available.Valid starting Expires Service principal 04/15/11
01:14:53 04/15/11 11:14:53 krbtgt/[email protected] renew until 04/16/11 01:14:53, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1 04/15/11 01:14:53 04/15/11 11:14:53 nfs/[email protected] renew until 04/16/11 01:14:53, Etype (skey, tkt): des-cbc-crc, aes256-cts-hmac-sha1-96So right now I am little bit stumped
* why another enctype is used
* why it does not succeed with the new enctypeBoth machines are running kernel 2.6.38 on debian sid with the latest Kerberos 1.9
If you need more information please tell me.
Kind regards,
Michael Guntsche--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
SGkgZ3V5cywKCk9uIEZyaSwgMjAxMS0wNC0xNSBhdCAwNjowMiArMDIwMCwgVmxhZGltaXIgRWxp c3NlZXYgd3JvdGU6Cj4gTWljaGFlbCwKPiAKPiBJJ3ZlIHBvc3RlZCBhbiBlbWFpbCB3aXRoIHRo ZSBzYW1lIHF1ZXN0aW9uIGhlcmUgYSBmZXcgd2Vla3MgYWdvLCBidXQgSQo+IGhhdmVuJ3QgYmVl biBhYmxlIHRvIGludmVzdGlnYXRlIHdoYXQgdGhlIHByb2JsZW0gaXMuIFRoZSBvbmx5Cj4gZGlm ZmVyZW5jZSBpbiBteSBzZXR1cCwgdGhhdCBzZXJ2ZXJzIGhhdmUga2VybmVsIDIuNi4zNiB0aGUg cmVzdCBpcwo+IGV4YWN0bHkgdGhlIHNhbWUgYW5kIGV4YWN0bHkgdGhlIHNhbWUgZXJyb3IuIFNl ZSB0aHJlYWQgd2l0aCB0aGUgc3ViamVjdAo+ICJycGMuc3ZjZ3NzZCBwcm9ibGVtIGFmdGVyIHVw ZGF0aW5nIGNsaWVudCAxLjIuMi0+MS4yLjMiLgoKSSB0aGluayB0aGUgcHJvYmxlbSBoZXJlIGlz IHRoZSBzYW1lIG9uZSB3ZSB3ZXJlIGV4cGVyaWVuY2luZywgd2l0aApsYXJnZSBncm91cCBtZW1i ZXJzaGlwcyBjYXVzaW5nIGludmFsaWQgd3JpdGVzIHRvIHRoZSBwcm9jZnMgcnBjIGNoYW5uZWwK ZmlsZXMuICBJZiB5b3Ugc3RyYWNlIHJwYy5zdmNnc3NkIChvciBycGMubW91bnRkLCB5b3Ugc2hv dWxkIHNlZSBpdAp0aGVyZSB0b28gSSB0aGluayksIHlvdSBzaG91bGQgc2VlIHRoZSB3cml0ZSgp IGNhbGxzIGJlaW5nIHNwbGl0IGludG8KbXVsdGlwbGUgMTAyNC1ieXRlIHNpemVkIHdyaXRlcywg ZWFjaCBvZiB3aGljaCBnZXQgRUlOVkFMIChvciAtRUlOVkFMLCBJCmZvcmdldCkgcmV0dXJuZWQu CgpJJ3ZlIHN1Ym1pdHRlZCBhIHBhdGNoIGZvciB0aGF0IGxhc3Qgd2Vlaywgd2hpY2ggaXMgd2Fp dGluZyBvbgppbmNsdXNpb24uICBJZiB5b3Ugd2FudCB0byB2ZXJpZnksIHBsZWFzZSBsb29rIGJh Y2sgaW4gdGhlIGFyY2hpdmUgYW5kCnRyeSB0aGUgcGF0Y2ggc3VibWl0dGVkIGluIHRoZSBtYWls IHdpdGggc3ViamVjdDoKCltQQVRDSCB2Ml0gbmZzLXV0aWxzOiBJbmNyZWFzZSB0aGUgc3RkaW8g ZmlsZSBidWZmZXIgc2l6ZSBmb3IgcHJvY2ZzCmZpbGVzCgpJdCBzZWVtcyB0byB3b3JrIGhlcmUs IEknZCBhcHByZWNpYXRlIGlmIHlvdSBjb3VsZCB0cnkgaXQgb3V0IGFuZCBsZXQgbWUKa25vdyBp ZiBpdCB3b3JrcyBmb3IgeW91IHRvbyA6KQoKCkJSCglTZWFuCgTvv717Lm7vv70r77+977+977+9 77+977+977+977+9KyXvv73vv71sendt77+977+9Yu+/veunsu+/ve+/vXLvv73vv716WO+/ve+/ vRnfsinvv73vv73vv713Kh9qZ++/ve+/ve+/vR7vv73vv73vv73vv73vv73domov77+977+977+9 eu+/vd6W77+977+9Mu+/vd6Z77+977+977+9Ju+/vSnfoe+/vWHvv73vv71/77+977+9Hu+/vUfv v73vv73vv71o77+9D++/vWo6K3bvv73vv73vv71377+92aU=
Sean,
I'm pretty sure this is not the case for my environment, as we have less then 100 groups in total.
Regards,
Vladimir.
Quoting "Finney, Sean" <[email protected]>:
Quoted 32 lines
Hi guys,
Quoted 7 lines · Vladimir Elisseev
Michael,
I've posted an email with the same question here a few weeks ago, but I haven't been able to investigate what the problem is. The only difference in my setup, that servers have kernel 2.6.36 the rest is exactly the same and exactly the same error. See thread with the subject "rpc.svcgssd problem after updating client 1.2.2->1.2.3".
I think the problem here is the same one we were experiencing, with large group memberships causing invalid writes to the procfs rpc channel files. If you strace rpc.svcgssd (or rpc.mountd, you should see it there too I think), you should see the
write()calls being split into multiple 1024-byte sized writes, each of which get EINVAL (or-EINVAL, I forget) returned.I've submitted a patch for that last week, which is waiting on inclusion. If you want to verify, please look back in the archive and try the patch submitted in the mail with subject:
[PATCH v2] nfs-utils: Increase the stdio file buffer size for procfs files
It seems to work here, I'd appreciate if you could try it out and let me know if it works for you too :)
BR
Sean
SGkgVmxhZGltaXIsCgpPbiBGcmksIDIwMTEtMDQtMTUgYXQgMDk6MTUgKzAyMDAsIFZsYWRpbWly IEVsaXNzZWV2IHdyb3RlOgo+IEknbSBwcmV0dHkgc3VyZSB0aGlzIGlzIG5vdCB0aGUgY2FzZSBm b3IgbXkgZW52aXJvbm1lbnQsIGFzIHdlIGhhdmUgIAo+IGxlc3MgdGhlbiAxMDAgZ3JvdXBzIGlu IHRvdGFsLgoKUmUtcmVhZGluZyB0aGUgbWFpbCwgSSByZWFsaXplIHlvdSBtaWdodCBiZSByaWdo dC4gIFRoZSBFSU5WQUxJRCBvbiB0aGUKcnBjIGNoYW5uZWwgZmlsZSBqdW1wZWQgb3V0IGF0IG1l IHRob3VnaCBzaW5jZSBpdCB3YXMgdGhlIHNhbWUgc3ltcHRvbQppbiBvdXIgY2FzZS4gIEFsc28s IGluIG91ciBjYXNlIHdlIHNhdyB0aGUgcHJvYmxlbSBpbiBhbGwgdmVyc2lvbnMgb2YKbmZzLXV0 aWxzIGFuZCBub3QganVzdCB0aGUgbGF0ZXN0IG9uZSwgc28gaXQgY291bGQgYmUgdGhhdCBpdCdz IGFuIGVycm9yCnNpbWlsYXIgaW4gbmF0dXJlIGJ1dCBub3QgdGhlIHNhbWUgdW5kZXJseWluZyBj YXVzZS4KCkFueXdheSwgaW4gY2FzZSBpdCdzIGFueSBoZWxwIChhc3N1bWluZyB0aGF0IHRoZSBw YXRjaCBkb2Vzbid0IGZpeCB0aGUKcHJvYmxlbSBmb3IgeW91KSwgdGhlIGtlcm5lbCByZXR1cm5z IEVJTlZBTElEIG9uIHdyaXRlKCkncyB0byB0aGUgZmlsZQp3aGVuIGFueSBzaW5nbGUgd3JpdGUg aXMgbm90IHByb3Blcmx5IGZvcm1hdHRlZCBhbmQgbmV3bGluZSB0ZXJtaW5hdGVkLgpJdCBzZWVt cyB0byBiZSB2ZXJ5IHBpY2t5IGFib3V0IHRoYXQuICBJZiB5b3Ugc3RyYWNlIC1zMjA0OCB0aGUK b2ZmZW5kaW5nIGRhZW1vbiwgaXQgc2hvdWxkIGJlIHByZXR0eSBjbGVhciB0aG91Z2gsIGVpdGhl ciB5b3UnbGwgc2VlIGEKc2VyaWVzIG9mIHNwbGl0IHVwIEJVRlNaIHNpemVkIHdyaXRlcyAoZWFj aCBnZXR0aW5nIEVJTlZBTElEKSBvciB5b3UnbGwKc2VlIGEgd3JpdGUgb3BlcmF0aW9uIHRoYXQn cyBvdGhlcndpc2Ugc29tZWhvdyBpbXByb3Blcmx5IGZvcm1hdHRlZC4KCkhvcGUgdGhhdCdzIGF0 IGxlYXN0IGEgYml0IGhlbHBmdWwsCgoJU2VhbgoT77+977+97Lm7HO+/vSbvv71+77+9Ju+/vRjv v73vv70rLe+/ve+/vd22F++/ve+/vXfvv73vv73Lm++/ve+/ve+/vW3vv71i77+977+9Z37Ip++/ vRfvv73vv73cqH3vv73vv73vv73GoHrvv70majordu+/ve+/ve+/vQfvv73vv73vv73vv716Wivv v73vv70rembvv73vv73vv71o77+977+977+9fu+/ve+/ve+/ve+/vWnvv73vv73vv71677+9Hu+/ vXfvv73vv73vv70/77+977+977+977+9Ju+/vSnfohtm
Quoted 11 lines · Sean
Hi guys,
Quoted 7 lines · Vladimir Elisseev
Michael,
I've posted an email with the same question here a few weeks ago, but I haven't been able to investigate what the problem is. The only difference in my setup, that servers have kernel 2.6.36 the rest is exactly the same and exactly the same error. See thread with the subject "rpc.svcgssd problem after updating client 1.2.2->1.2.3".
Quoted 6 lines
I think the problem here is the same one we were experiencing, with large group memberships causing invalid writes to the procfs rpc channel files. If you strace rpc.svcgssd (or rpc.mountd, you should see it there too I think), you should see the
write()calls being split into multiple 1024-byte sized writes, each of which get EINVAL (or-EINVAL, I forget) returned.
<snip>
Thank you for the information, but I got it working in the meantime. The main problem still is that the code for some reason tries to use AES although I tried specifying a different enctype in my kerberos config. Nevertheless it should just work with AES as well, so where was the problem?
Quite simple....missing kernel support. I enabled AES support but I DID NOT enable CTS support which is of course needed as well. So after compiling the server and client kernels with BOTH AES and CTS support I can no mount the NFS4 export without any issues.
I still think the error message could be a little bit more verbose/meaningful like "Missing kernel support for X" instead of a rather cryptic message.
Once again, sorry for the noise and I'll keep testing this setup now more often so this hopefully does not happen again.
Kind regards,
Michael Guntsche
Quoted 9 lines · Michael Guntsche
Thank you for the information, but I got it working in the meantime. The main problem still is that the code for some reason tries to use AES although I tried specifying a different enctype in my kerberos config. Nevertheless it should just work with AES as well, so where was the problem?
Quite simple....missing kernel support. I enabled AES support but I DID NOT enable CTS support which is of course needed as well. So after compiling the server and client kernels with BOTH AES and CTS support I can no mount the NFS4 export without any issues.
Sigh. We really should not allow that kind of config. It just creates confusion.
Kevin, what are the dependencies for the kerberos V module today? Am I missing something in the following list?
depends on SUNRPC && CRYPTO
depends on CRYPTO_MD5 && CRYPTO_DES && CRYPTO_CBC && CRYPTO_CTS
depends on CRYPTO_ECB && CRYPTO_HMAC && CRYPTO_MD5 &&
CRYPTO_SHA1
depends on CRYPTO_AESCheers
Trond
Quoted 14 lines · Trond Myklebust
Quoted 9 lines · Michael Guntsche
Thank you for the information, but I got it working in the meantime. The main problem still is that the code for some reason tries to use AES although I tried specifying a different enctype in my kerberos config. Nevertheless it should just work with AES as well, so where was the problem?
Quite simple....missing kernel support. I enabled AES support but I DID NOT enable CTS support which is of course needed as well. So after compiling the server and client kernels with BOTH AES and CTS support I can no mount the NFS4 export without any issues.Sigh. We really should not allow that kind of config. It just creates confusion.
Quoted 6 lines
Kevin, what are the dependencies for the kerberos V module today? Am I missing something in the following list?
depends on SUNRPC && CRYPTO depends on CRYPTO_MD5 && CRYPTO_DES && CRYPTO_CBC && CRYPTO_CTS depends on CRYPTO_ECB && CRYPTO_HMAC && CRYPTO_MD5 &&
I suppose we only need one test for CRYPTO_MD5...
Quoted 10 lines
CRYPTO_SHA1
depends onCRYPTO_AESCheers
Trond--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 17 lines · Trond Myklebust
Quoted 9 lines · Michael Guntsche
Thank you for the information, but I got it working in the meantime. The main problem still is that the code for some reason tries to use AES although I tried specifying a different enctype in my kerberos config. Nevertheless it should just work with AES as well, so where was the problem?
Quite simple....missing kernel support. I enabled AES support but I DID NOT enable CTS support which is of course needed as well. So after compiling the server and client kernels with BOTH AES and CTS support I can no mount the NFS4 export without any issues.Sigh. We really should not allow that kind of config. It just creates confusion.
Kevin, what are the dependencies for the kerberos V module today? Am I missing something in the following list?
Quoted 8 lines
depends on SUNRPC && CRYPTO
depends onCRYPTO_MD5&&CRYPTO_DES&&CRYPTO_CBC&&CRYPTO_CTSdepends onCRYPTO_ECB&&CRYPTO_HMAC&&CRYPTO_MD5&&CRYPTO_SHA1
depends onCRYPTO_AESCheers
Trond
Yeah, I think that stuff got left out of the final patches.
DES3 needs (in addition to the stuff already there for DES) HMAC and SHA1 AES needs SHA1 AES CTS
RC4 needs ECB ARC4 MD5
So I think you are only missing CRYPTO_ARC4.