Skip to contentSkip to conversation

svcgssd always sets an infinite expiry on authentication tokens etc.

PATCH/RFC Neil Brown 5 participants 6 messages Dec 2, 2008 – Dec 3, 2008 4 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Neil Brown

author
#0 lore raw

Hi,
I have a report of an NFS server which runs out of kernel memory when it gets heave rpcsec_gss traffic (auth_sys doesn't trigger the problem so it must be gss related).

From looking at /proc/slab_allocators it seems that the main user of memory is the rsc and rsi caches.
It appears entries are inserted into these caches with an expiry of 'forever' so they grow but never shrink.
We should fix this.

For the rsi (init) cache I assume the entry is only needed once so a short expiry of (say) one minute should be plenty.
For the rsc (context) cache, the entry could be needed repeatedly during the lifetime of a 'session'. However eventually it will become stale and should be allowed to expire.

I assume that if the kernel requests a particular entry a second time, an hour later, it will get the same answer - is that correct?

In that case, setting the expiry to something largish seems appropriate.

Hence the following patch (untested yet - but I will get it tested in due course).

Does this seem reasonable?

Thanks,
NeilBrown

utils/gssd/svcgssd_proc.c +5 -2
index 794c2f4..088a007 100644
@@ -86,7 +86,9 @@ do_svc_downcall(gss_buffer_desc *out_handle, struct svc_cred *cred,
 	}
 	qword_printhex(f, out_handle->value, out_handle->length);
 	/* XXX are types OK for the rest of this? */
-	qword_printint(f, 0x7fffffff); /*XXX need a better timeout */
+
+	/* 'context' could be needed for a while. */
+	qword_printint(f, time(0) + 60*60);
 	qword_printint(f, cred->cr_uid);
 	qword_printint(f, cred->cr_gid);
 	qword_printint(f, cred->cr_ngroups);
@@ -130,7 +132,8 @@ send_response(FILE *f, gss_buffer_desc *in_handle, gss_buffer_desc *in_token,
 
 	qword_addhex(&bp, &blen, in_handle->value, in_handle->length);
 	qword_addhex(&bp, &blen, in_token->value, in_token->length);
-	qword_addint(&bp, &blen, 0x7fffffff); /*XXX need a better timeout */
+	/* INIT context info will only be needed for a short while */
+	qword_addint(&bp, &blen, time(0) + 60);
 	qword_adduint(&bp, &blen, maj_stat);
 	qword_adduint(&bp, &blen, min_stat);
 	qword_addhex(&bp, &blen, out_handle->value, out_handle->length);

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hi Neil,
This seems reasonable.

I have a patch somewhere that gets the actual Kerberos expiration that could be used for the rsc timeout. But I think this should be fine for now. (Perhaps at the cost of requiring clients to negotiate a new context every hour?)

K.C.

Quoted 65 lines · Neil Brown

Hi,
I have a report of an NFS server which runs out of kernel memory when it gets heave rpcsec_gss traffic (auth_sys doesn't trigger the problem so it must be gss related).

From looking at /proc/slab_allocators it seems that the main user of memory is the rsc and rsi caches.
It appears entries are inserted into these caches with an expiry of 'forever' so they grow but never shrink.
We should fix this.

For the rsi (init) cache I assume the entry is only needed once so a short expiry of (say) one minute should be plenty.
For the rsc (context) cache, the entry could be needed repeatedly during the lifetime of a 'session'. However eventually it will become stale and should be allowed to expire.

I assume that if the kernel requests a particular entry a second time, an hour later, it will get the same answer - is that correct?

In that case, setting the expiry to something largish seems appropriate.

Hence the following patch (untested yet - but I will get it tested in due course).

Does this seem reasonable?

Thanks,
NeilBrown

utils/gssd/svcgssd_proc.c +5 -3
index 794c2f4..088a007 100644
@@ -86,7 +86,9 @@ do_svc_downcall(gss_buffer_desc *out_handle, struct svc_cred *cred,
       }
       qword_printhex(f, out_handle->value, out_handle->length);
       /* XXX are types OK for the rest of this? */
-       qword_printint(f, 0x7fffffff); /*XXX need a better timeout */
+
+       /* 'context' could be needed for a while. */
+       qword_printint(f, time(0) + 60*60);
       qword_printint(f, cred->cr_uid);
       qword_printint(f, cred->cr_gid);
       qword_printint(f, cred->cr_ngroups);
@@ -130,7 +132,8 @@ send_response(FILE *f, gss_buffer_desc *in_handle, gss_buffer_desc *in_token,
 
       qword_addhex(&bp, &blen, in_handle->value, in_handle->length);
       qword_addhex(&bp, &blen, in_token->value, in_token->length);
-       qword_addint(&bp, &blen, 0x7fffffff); /*XXX need a better timeout */
+       /* INIT context info will only be needed for a short while */
+       qword_addint(&bp, &blen, time(0) + 60);
       qword_adduint(&bp, &blen, maj_stat);
       qword_adduint(&bp, &blen, min_stat);
       qword_addhex(&bp, &blen, out_handle->value, out_handle->length);
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4d569c330812012145y2353bc9asd7a0c62fef42ed3a-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#4 lore raw
Quoted 7 lines · Kevin Coffman

Hi Neil,
This seems reasonable.

I have a patch somewhere that gets the actual Kerberos expiration that could be used for the rsc timeout. But I think this should be fine for now. (Perhaps at the cost of requiring clients to negotiate a new context every hour?)

This question is a bit worrisome, imho... I understand the need to release memory consumed by dead contexts but on the other hand, renegotiating contexts every hour on the hours seems a bit costly as well...

Does it make sense to make this time out configurable? Yes, it would be a very obscure knob, but in the unlikely event there is a bug in the renegotiating code or renegotiating simply becomes too costly, I think it would good to have a way to dial the time out back up as a work-around.

steved.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 12 lines · Steve Dickson
Quoted 7 lines · Kevin Coffman

Hi Neil,
This seems reasonable.

I have a patch somewhere that gets the actual Kerberos expiration that could be used for the rsc timeout. But I think this should be fine for now. (Perhaps at the cost of requiring clients to negotiate a new context every hour?)

This question is a bit worrisome, imho... I understand the need to release memory consumed by dead contexts but on the other hand, renegotiating contexts every hour on the hours seems a bit costly as well...

Quoted 6 lines

Does it make sense to make this time out configurable? Yes, it would be a very obscure knob, but in the unlikely event there is a bug in the renegotiating code or renegotiating simply becomes too costly, I think it would good to have a way to dial the time out back up as a work-around.

steved.

Hi Steve,

Rather than adding another config knob, how 'bout I dust off my patch to get the "right" timeout value? I should be able to make that available tomorrow. (I have limited resources to work on this at the moment.)

K.C.

A message that is not in the archive (sent off-list or not yet received): 4d569c330812020940n3b8561fexfb97d89a7d5779a4-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#8 lore raw
Quoted 21 lines

Dec 2, 2008 at 12:40 PM, Kevin Coffman <[email protected]> wrote:

Quoted 12 lines · Steve Dickson
Quoted 7 lines · Kevin Coffman

Hi Neil,
This seems reasonable.

I have a patch somewhere that gets the actual Kerberos expiration that could be used for the rsc timeout. But I think this should be fine for now. (Perhaps at the cost of requiring clients to negotiate a new context every hour?)

This question is a bit worrisome, imho... I understand the need to release memory consumed by dead contexts but on the other hand, renegotiating contexts every hour on the hours seems a bit costly as well...

Quoted 6 lines

Does it make sense to make this time out configurable? Yes, it would be a very obscure knob, but in the unlikely event there is a bug in the renegotiating code or renegotiating simply becomes too costly, I think it would good to have a way to dial the time out back up as a work-around.

steved.

Quoted 8 lines

Hi Steve,

Rather than adding another config knob, how 'bout I dust off my patch to get the "right" timeout value? I should be able to make that available tomorrow. (I have limited resources to work on this at the moment.)

K.C.

It took me a bit longer than I anticipated to retrofit my changes for this. I have patches that I will send out for review later tonight or tomorrow.

K.C.

#9 lore raw
In reply to a message not in the archive
Quoted 9 lines · Neil Brown

I have a report of an NFS server which runs out of kernel memory when it gets heave rpcsec_gss traffic (auth_sys doesn't trigger the problem so it must be gss related).

From looking at /proc/slab_allocators it seems that the main user of memory is the rsc and rsi caches.
It appears entries are inserted into these caches with an expiry of 'forever' so they grow but never shrink.
We should fix this.

Yes, definitely a leak. But it's funny, I have a strong memory of a conversation on one of these lists with someone who looked into this problem and found that even with some silly artificial tests that established as many contexts per second as possible, they weren't able to see significant memory consumption--but I can't find the thread now.

Some limit on the size might also be nice depending on what the worst case looks like with expiries of about a day.

--b.

Quoted 17 lines

For the rsi (init) cache I assume the entry is only needed once so a short expiry of (say) one minute should be plenty.
For the rsc (context) cache, the entry could be needed repeatedly during the lifetime of a 'session'. However eventually it will become stale and should be allowed to expire.

I assume that if the kernel requests a particular entry a second time, an hour later, it will get the same answer - is that correct?

In that case, setting the expiry to something largish seems appropriate.

Hence the following patch (untested yet - but I will get it tested in due course).

Does this seem reasonable?