Skip to contentSkip to conversation

NFSD over TCP: TCP broken?

DISCUSSION Shirish Kalele 2 participants 2 messages Oct 16, 2001 – Oct 17, 2001

lore
Share & tools
lore
#0 lore raw
Quoted 9 lines

Hi,

I've been looking at running nfsd over tcp on Linux. I modified the #ifdef so that nfsd uses tcp. I also made writes to the socket blocking, so that the thread blocks till the entire reply has been accepted by TCP. (I know the right way is going to be to have an independent thread whose job would be to just pick replies off a queue and block on sending them to tcp, but this is what I've done temporarily.)

Quoted 5 lines

Then I tried to copy a directory from a Solaris client to the Linux server using nfsv3 over tcp. This took a long time, with lots of delays where nothing was being transferred.

Looking at the network traces, it looks like the RPC records being sent

over

Quoted 2 lines

TCP are inconsistent with the lengths specified in the record marker. This happens mainly when 3-4 requests arrive one after the other and you have

3-4

Quoted 2 lines

threads replying to these requests in parallel. It looks like TCP gets hopelessly confused and botches up the replies being sent. I point my

finger

Quoted 1 line

at TCP because tcp_sendmsg returns a valid length indicating that the

entire

Quoted 2 lines

reply was accepted, but the tcp sequence numbers show that the RPC record sent on the wire wasn't equal to the length accepted by TCP. After a

while,

Quoted 5 lines

the client realizes it's out of sync when it gets an invalid RPC record marker, and resets and reconnects. This repeats multiple times.

Is TCP known to break when multiple threads try to send data down the pipe simulaneously? Is there a known fix for this? Where should I be focussing

to

Quoted 8 lines

fix the problem?

I'm not on the list, so please include me in replies.

Thanks,
Shirish

#1 lore raw
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Shirish Kalele <[email protected]> writes:

Hi,

Looking at the network traces, it looks like the RPC records being sent

> over
>> TCP are inconsistent with the lengths specified in the record >> marker. This happens mainly when 3-4 requests arrive one after >> the other and you have
> 3-4
>> threads replying to these requests in parallel. It looks like >> TCP gets hopelessly confused and botches up the replies being >> sent. I point my
> finger
>> at TCP because tcp_sendmsg returns a valid length indicating >> that the
> entire
>> reply was accepted, but the tcp sequence numbers show that the >> RPC record sent on the wire wasn't equal to the length accepted >> by TCP. After a
> while,
>> the client realizes it's out of sync when it gets an invalid >> RPC record marker, and resets and reconnects. This repeats >> multiple times.

This is normal. Nobody has fixed the RPC server code. There are plenty of possible sources of the above problem, but my main suspect is the fact that the TCP reply code uses blocking socket operations (will change once we actually go in for supporting TCP), but doesn't provide any mechanism for preventing another thread from using the socket while the original writer is sleeping.

Fix: Set up a semaphore in the struct svc_sock somewhere, and use it to gate write acces to the socket...

Cheers,
Trond