gssd: cut down log chatter in gssd_search_krb5_keytab
Subsystems (2)
PATCH 1/1 gssd: cut down log chatter in gssd_search_krb5_keytab
Because the kernel now looks for krb5 creds by default when mounting, we need to always have clients run rpc.gssd to avoid the 15s hang on the first mount attempt.
In situations however where people don't have krb5 set up, this leads to a lot of log spamming on mount attempts. Try to cut down on some of the log chatter by lowering the priority of the log messages in gssd_search_krb5_keytab().
--- utils/gssd/krb5_util.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
utils/gssd/krb5_util.c +4 -4
@@ -711,13 +711,13 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,*/ if ((code =krb5_kt_get_name (context, kt, kt_name,BUFSIZ ))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s attempting to get keytab name\n" , k5err); +printerr (1 ,"ERROR: %s attempting to get keytab name\n" , k5err); retval = code; goto out; }if ((code =krb5_kt_start_seq_get (context, kt, &cursor))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s while beginning keytab scan " +printerr (1 ,"ERROR: %s while beginning keytab scan " "for keytab '%s'\n" , k5err, kt_name); retval = code; goto out;
@@ -727,7 +727,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_unparse_name (context, kte->principal, &pname))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: Skipping keytab entry because " +printerr (1 ,"WARNING: Skipping keytab entry because " "we failed to unparse principal name: %s\n" , k5err);k5_free_kt_entry (context, kte);
@@ -768,7 +768,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_kt_end_seq_get (context, kt, &cursor))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: %s while ending keytab scan for " +printerr (1 ,"WARNING: %s while ending keytab scan for " "keytab '%s'\n" , k5err, kt_name); }
Signature · 1 line
1.8.3.1
Quoted 3 lines · Jeff Layton
Because the kernel now looks for krb5 creds by default when mounting, we need to always have clients run rpc.gssd to avoid the 15s hang on the first mount attempt.
Not sure if it's the same hangs, but I see about the same length of a hang on my Debian wheezy systems when accessing the first file for a data operation, with rpc.gssd running or not. Funnily enough this also happens when trying to read a non-existand file, but not for things like a simple readdir.
Still looking into it, but being a non-interruptible sleep it's fairly annoying.
That being said Debian wheezy also only starts rpc.gssd from the init scripts if it sees a krb option in /etc/fstab, which I don't have.
Quoted 11 lines · On Thu, 7 Nov 2013 07:58:37 -0800 Christoph Hellwig <[email protected]> wrote:
Quoted 3 lines · Jeff Layton
Because the kernel now looks for krb5 creds by default when mounting, we need to always have clients run rpc.gssd to avoid the 15s hang on the first mount attempt.
Not sure if it's the same hangs, but I see about the same length of a hang on my Debian wheezy systems when accessing the first file for a data operation, with rpc.gssd running or not. Funnily enough this also happens when trying to read a non-existand file, but not for things like a simple readdir.
Quoted 6 lines
Still looking into it, but being a non-interruptible sleep it's fairly annoying.
That being said Debian wheezy also only starts rpc.gssd from the init scripts if it sees a krb option in
/etc/fstab, which I don't have.
It may be the same issue. IIRC, the attempt to use krb5i by default is done in the SETCLIENTID call (see kernel commit 4edaa30888 for some background, but there were some later tweaks to this approach).
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 3 lines · Jeff Layton
It may be the same issue. IIRC, the attempt to use krb5i by default is done in the SETCLIENTID call (see kernel commit
4edaa30888for some background, but there were some later tweaks to this approach).
I'm still seing my issue after reverting that commit by hand.
Quoted 6 lines · On Thu, 7 Nov 2013 08:21:23 -0800 Christoph Hellwig <[email protected]> wrote:
Quoted 3 lines · Jeff Layton
It may be the same issue. IIRC, the attempt to use krb5i by default is done in the SETCLIENTID call (see kernel commit
4edaa30888for some background, but there were some later tweaks to this approach).I'm still seing my issue after reverting that commit by hand.
Hrm...
You'd also see these messages in the log if it were the same problem:
RPC: AUTH_GSS upcall timed out.
...it could also be a delay when trying to set up the callback channel if you're running v4.0. Do you have the callback port firewalled off?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 10 lines · Jeff Layton
Quoted 1 line
I'm still seing my issue after reverting that commit by hand.
Hrm...
You'd also see these messages in the log if it were the same problem:
RPC:
AUTH_GSSupcall timed out....it could also be a delay when trying to set up the callback channel if you're running v4.0. Do you have the callback port firewalled off?
Not seeing any of those, and running with 4.1 at the moment, although I could try 4.0. Talking to a server on the same host, so firewalling is unlikely.
Quoted 7 lines · On Thu, 7 Nov 2013 08:35:20 -0800 Christoph Hellwig <[email protected]> wrote:
Quoted 6 lines · Jeff Layton
Quoted 1 line
I'm still seing my issue after reverting that commit by hand.
Hrm...
You'd also see these messages in the log if it were the same problem:
Quoted 9 lines
Quoted 4 lines
RPC:
AUTH_GSSupcall timed out....it could also be a delay when trying to set up the callback channel if you're running v4.0. Do you have the callback port firewalled off?
Not seeing any of those, and running with 4.1 at the moment, although I could try 4.0. Talking to a server on the same host, so firewalling is unlikely.
No idea then, sorry...
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 1 line · Jeff Layton
No idea then, sorry...
4.0 is also affected, interestingly it's only reproducible on the first mount of a volume after the server started, so I'll assume a server issue. I'm trying to drill it down time permitting, but so far I have no clue what's going on. Doesn't help that recent versions of latencytop are unusable..
Hello,
My thoughts...
Quoted 24 lines · Jeff Layton
Because the kernel now looks for krb5 creds by default when mounting, we need to always have clients run rpc.gssd to avoid the 15s hang on the first mount attempt.
In situations however where people don't have krb5 set up, this leads to a lot of log spamming on mount attempts. Try to cut down on some of the log chatter by lowering the priority of the log messages in
gssd_search_krb5_keytab().Signed-off-by Jeff Layton <[email protected]>--- utils/gssd/krb5_util.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)utils/gssd/krb5_util.c +1
-1@@ -711,13 +711,13 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,*/ if ((code =krb5_kt_get_name (context, kt, kt_name,BUFSIZ ))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s attempting to get keytab name\n" , k5err); +printerr (1 ,"ERROR: %s attempting to get keytab name\n" , k5err);
utils/gssd/krb5_util.c gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt, · line 723
Quoted 10 lines
utils/gssd/krb5_util.c +1
-1retval = code; goto out; }if ((code =krb5_kt_start_seq_get (context, kt, &cursor))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s while beginning keytab scan " +printerr (1 ,"ERROR: %s while beginning keytab scan " "for keytab '%s'\n" , k5err, kt_name); retval = code; goto out;
I'm thinking these need to stay... I don't think is a good idea to mask out messages describing failures...
Quoted 10 lines
utils/gssd/krb5_util.c +1
-1@@ -727,7 +727,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_unparse_name (context, kte->principal, &pname))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: Skipping keytab entry because " +printerr (1 ,"WARNING: Skipping keytab entry because " "we failed to unparse principal name: %s\n" , k5err);k5_free_kt_entry (context, kte);
utils/gssd/krb5_util.c gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt, · line 773
Quoted 7 lines
utils/gssd/krb5_util.c +1
-1@@ -768,7 +768,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_kt_end_seq_get (context, kt, &cursor))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: %s while ending keytab scan for " +printerr (1 ,"WARNING: %s while ending keytab scan for " "keytab '%s'\n" , k5err, kt_name); }
I think warnings would be ok to mask out...
steved.
Quoted 2 lines
Quoted 33 lines · On Thu, 07 Nov 2013 11:47:55 -0500 Steve Dickson <[email protected]> wrote:
Hello,
My thoughts...
Quoted 28 lines · Jeff Layton
Because the kernel now looks for krb5 creds by default when mounting, we need to always have clients run rpc.gssd to avoid the 15s hang on the first mount attempt.
In situations however where people don't have krb5 set up, this leads to a lot of log spamming on mount attempts. Try to cut down on some of the log chatter by lowering the priority of the log messages in
gssd_search_krb5_keytab().Signed-off-by Jeff Layton <[email protected]>--- utils/gssd/krb5_util.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)utils/gssd/krb5_util.c +1
-1@@ -711,13 +711,13 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,*/ if ((code =krb5_kt_get_name (context, kt, kt_name,BUFSIZ ))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s attempting to get keytab name\n" , k5err); +printerr (1 ,"ERROR: %s attempting to get keytab name\n" , k5err); retval = code; goto out; }if ((code =krb5_kt_start_seq_get (context, kt, &cursor))) {
Quoted 9 lines
utils/gssd/krb5_util.c gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt, · line 723
Quoted 6 lines
utils/gssd/krb5_util.c +1
-1k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"ERROR: %s while beginning keytab scan " +printerr (1 ,"ERROR: %s while beginning keytab scan " "for keytab '%s'\n" , k5err, kt_name); retval = code; goto out;I'm thinking these need to stay... I don't think is a good idea to mask out messages describing failures...
The above message is the one that's the problem when there is no keytab. I just went ahead and lowered the priority on the others for good measure. If you don't want to take the above hunk, then there's probably no need to bother with this patch at all.
Quoted 23 lines
Quoted 10 lines
utils/gssd/krb5_util.c +1
-1@@ -727,7 +727,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_unparse_name (context, kte->principal, &pname))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: Skipping keytab entry because " +printerr (1 ,"WARNING: Skipping keytab entry because " "we failed to unparse principal name: %s\n" , k5err);k5_free_kt_entry (context, kte);utils/gssd/krb5_util.c gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt, · line 773
Quoted 7 lines
utils/gssd/krb5_util.c +1
-1@@ -768,7 +768,7 @@ gssd_search_krb5_keytab(krb5_context context, krb5_keytab kt,if ((code =krb5_kt_end_seq_get (context, kt, &cursor))) { k5err =gssd_k5_err_msg (context, code); -printerr (0 ,"WARNING: %s while ending keytab scan for " +printerr (1 ,"WARNING: %s while ending keytab scan for " "keytab '%s'\n" , k5err, kt_name); }I think warnings would be ok to mask out...
steved.
Quoted 2 lines
Signature · 1 line
Jeff Layton <[email protected]>