Skip to contentSkip to conversation

task->tk_timeout = 0

DISCUSSION Olaf Kirch 2 participants 2 messages Feb 26, 2004

Subsystems (2)
lore
Share & tools
lore

Olaf Kirch

author
#0 lore raw

Hi all,

I've been debugging mysterious hangs in the NFS clients for a while now. The symptoms were always the same: NFS server goes away, comes back after a long time, but the mount point remains stuck.

When inspecting the list of RPC tasks, I often found that one process was on the pending queue (i.e. waiting for a reply) but task->tk_timeout was 0.

I added a couple of diagnostic printks, and one of them triggered yesterday, pointing to this piece of code in xprt_transmit:

if (!xprt->nocong) {
	int timer = task->tk_msg.rpc_proc->p_timer;
	timeout = rpc_calc_rto(clnt->cl_rtt, timer);
	timeout <<= rpc_ntimeo(clnt->cl_rtt, timer);
	timeout <<= clnt->cl_timeout.to_retries
		- req->rq_timeout.to_retries;
	if (timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
	else if (timeout == 0) {
		printk(KERN_ERR "RPC task timeout == 0, please tell okir\n");
		timeout = req->rq_timeout.to_maxval;
	}
	...
}

So apparently one of the shift operations above overflows. I suspect rpc_ntimeo because it can be arbitrarily large. But the ntimeouts value isn't updated until we've received a reply (is this intentional, Trond?)

So what would have to happen for this bug to trigger is

send request
server hangs
retransmit request 1000 times
server comes back
retransmit request
receive reply, set ntimeout = 1000

send another request. timeout overflows and becomes 0 request gets lost
task waits indefinitely.

I think the most robust way to fix this is simply this:

		- req->rq_timeout.to_retries;
	if (timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
-	if (timeout > req->rq_timeout.to_maxval)
+	if (!timeout || timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;

It may also make sense to clamp the ntimeouts value to something reasonable (e.g. 8)

Olaf

Signature · 13 lines
Olaf Kirch     |  Stop wasting entropy - start using predictable
[email protected]   |  tempfile names today!
---------------+ 


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw

På to , 26/02/2004 klokka 03:08, skreiv Olaf Kirch:

Quoted 1 line

So apparently one of the shift operations above overflows. I suspect

rpc_ntimeo because it can be arbitrarily large. But the ntimeouts value isn't updated until we've received a reply (is this intentional, Trond?)

Quoted 23 lines

So what would have to happen for this bug to trigger is

send request
server hangs
retransmit request 1000 times
server comes back
retransmit request
receive reply, set ntimeout = 1000

send another request. timeout overflows and becomes 0 request gets lost
task waits indefinitely.

I think the most robust way to fix this is simply this:

		- req->rq_timeout.to_retries;
	if (timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
(quoted diff) +1 -1
-	if (timeout > req->rq_timeout.to_maxval)
+	if (!timeout || timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
Quoted 1 line

It may also make sense to clamp the ntimeouts value to something

reasonable (e.g. 8)

Agreed, and in fact, I believe at some point it was clamped to 8. Thanks, and well spotted...

Cheers,
Trond

-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs