Skip to contentSkip to conversation

ncpfs and TCP vs UDP

DISCUSSION Pierre Ossman 3 participants 5 messages Jan 26, 2007 – Jan 27, 2007

lore
Share & tools
lore
#0 lore raw

Hi Petr,

I was hoping you could give me some input on another concern I have. Which of TCP and UDP is the preferred transport for NCP? The client for Windows seems to use TCP, which would suggest that that is the most tested dialect. I also did a quick test with bonnie++:

UDP:

Version 1.03 -----Sequential Output------ --Sequential Input- --Random-

             -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
        500M  1569  19  1714  10   826   9  1414  18  1569   9  65.2   3
             ------Sequential Create------ --------Random Create--------
             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
       files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
          16   460  14   897  16   556  15   491  15  1131  13   320   6

poseidon.drzeus.cx,500M,1569,19,1714,10,826,9,1414,18,1569,9,65.2,3,16,460,14,897,16,556,15,491,15,1131,13,320,6

TCP:

Version 1.03 -----Sequential Output------ --Sequential Input- --Random-

             -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
        500M  1653  19  3711  11  2312   9  1419  18  4489   6  91.5   1
             ------Sequential Create------ --------Random Create--------
             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
       files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
          16   483  14   912  15   587  14   483  14  1176  13   287   6

poseidon.drzeus.cx,500M,1653,19,3711,11,2312,9,1419,18,4489,6,91.5,1,16,483,14,912,15,587,14,483,14,1176,13,287,6

TCP has roughly twice the throughput of UDP. Note that this was done against a vmware machine that wasn't exactly maxing out the network.

Input welcome.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#1 lore raw

Quoting Pierre Ossman <[email protected]>:

Quoted 1 line

Hi Petr,

Hello,

Quoted 4 lines

I was hoping you could give me some input on another concern I have. Which of TCP and UDP is the preferred transport for NCP? The client for Windows seems to use TCP, which would suggest that that is the most tested dialect. I also did a quick test with bonnie++:

TCP is definitely preferred. There are couple of reasons why you should prefer TCP:

(1) There is server configuration option to disable NCP/UDP. You cannot disable NCP/TCP that easily.

(2) TCP (NCP over TCP) retransmits only missing data, and it can ask for retransmit much sooner as it knows what link latency is. NCP/UDP can only ask for complete packet retransmission, and it has no good idea what's link latency because there is no ACK from server when it receives request - you can only resend after usual link latency + time for process request, so you'll wait longer for retransmit, and on retransmit you need to send again complete request (which can be 64KB of data if you use 64KB buffer size...)

(3) To avoid problems with retransmits ncpfs uses default buffer size 60KB for TCP (SOCK_STREAM), while 1KB for UDP/IPX (it must be multiple of sector size, so using 1.4KB is not an option). So if you read 1 page, you get 1 request/reply when using TCP, but 4 requests/replies in UDP/IPX. And as all this is fully synchronous, and for today's link latency is dominating factor, it will take 4 times longer...

(4) Theoretically with TCP you should never need retransmits as TCP takes care of that. Unfortunately due to server implementation you still cannot have more than one request in flight (at least with NW5 - I never tried it with NW6 as spec says I should not do that, and NW5 implementation agreed with spec).

Quoted 6 lines

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

        500M  1569  19  1714  10   826   9  1414  18  1569   9  65.2   3
             ------Sequential Create------ --------Random Create--------
             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
       files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
          16   460  14   897  16   556  15   491  15  1131  13   320   6

What was 'Block' size ? For 1KB block size you should get more or less same speed for TCP and UDP. If you have link with big latency and big throughput, then for 2KB block TCP should be 2 times faster, for 60KB block 60 times faster, and above 60KB block difference should stay on 1:60.

You can try bumping NCP/UDP block size to 60KB as well, but my exprience with switches and NetWare is that in such cases you can get into situation with deterministic packet loss - like that from 40 UDP packets sent back to back to server every 12th gets lost - and if you resend, again 12th packet gets lost, again and again until you have some luck, or until client decides that server is dead.
Petr

#2 lore raw
Quoted 16 lines · On Jan 26 2007 22:20, [email protected] wrote:

Quoting Pierre Ossman <[email protected]>:

TCP is definitely preferred. There are couple of reasons why you should prefer TCP:

(1) There is server configuration option to disable NCP/UDP. You cannot disable NCP/TCP that easily.

(2) TCP (NCP over TCP) retransmits only missing data, and it can ask for retransmit much sooner as it knows what link latency is. NCP/UDP can only ask for complete packet retransmission, and it has no good idea what's link latency because there is no ACK from server when it receives request - you can only resend after usual link latency + time for process request, so you'll wait longer for retransmit, and on retransmit you need to send again complete request (which can be 64KB of data if you use 64KB buffer size...)

Quoted 6 lines

(3) To avoid problems with retransmits ncpfs uses default buffer size 60KB for TCP (SOCK_STREAM), while 1KB for UDP/IPX (it must be multiple of sector size, so using 1.4KB is not an option). So if you read 1 page, you get 1 request/reply when using TCP, but 4 requests/replies in UDP/IPX. And as all this is fully synchronous, and for today's link latency is dominating factor, it will take 4 times longer...

[and (4)]

Well, probably the same reason as NFS over UDP is discouraged. See nfs(5) section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)

	-`J'
#3 lore raw
Quoted 4 lines · Jan Engelhardt

Well, probably the same reason as NFS over UDP is discouraged. See nfs(5) section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)

I have no such warning in my nfs(5), but I am aware of this yes. Somewhat amusing that both nfs and ncpfs tend to default to using udp with this in mind. ;)

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#4 lore raw
Quoted 7 lines
Quoted 3 lines

Well, probably the same reason as NFS over UDP is discouraged. See nfs(5) section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)

I have no such warning in my nfs(5), but I am aware of this yes. Somewhat amusing that both nfs and ncpfs tend to default to using udp with this in mind. ;)

nfs has already been "fixed" to prefer tcp over udp (i.e. the default is tcp)

	-`J'