Skip to contentSkip to conversation

NFS performance tuning

DISCUSSION Stephen Carville 7 participants 12 messages Jan 6, 2006 – Jan 9, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have NFS server running on a Dell 2850:

Name: Amazon

   CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

The client is a Dell 2650 running Oracle 9i

Name: Tigris

   CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Both have hyperthreading enabled.
The machines are connected via a dedicated gigabit switch.

The goal (among others) is to run the nightly RMAN backups of the Oracle database to the NFS drive on Amazon where it is backed up to tape later. To this end I mount the NFS volume, run the backups, and unmount the volme.

That part works well. However the thruput on the connection is much less than I expected. Simple file transfers for example give only about:

Using NFS
Amazon to Tigris: 168 Mbps
Tigris to Amazon: 104 Mbps

Using ftp
ssh A to T: 320 Mbps
ssh T to A 360 Mbps

I can accept that I may never get a full Gigabit per second over copper but it still looks like there is room for improvement in the NFS trnsfer speeds.

Suggestions?

Signature · 17 lines
Stephen Carville <[email protected]>
Unix and Network Admin
Nationwide Totalflood
6033 W. Century Blvd
Los Angeles, CA 90045
310-342-3602



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 14 lines · Stephen Carville

I have NFS server running on a Dell 2850:

   Name: Amazon
    CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

Quoted 8 lines

The client is a Dell 2650 running Oracle 9i

   Name: Tigris
    CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Why are you using NFSv2? That forces the server to use synchronous writes (i.e. each write must be committed to disk before the server replies to the client) and is very inefficient.

If your application is write-intensive, you should always choose NFSv3 or greater.

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 28 lines · Trond Myklebust
Quoted 14 lines · Stephen Carville

I have NFS server running on a Dell 2850:

Name: Amazon

   CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

Quoted 10 lines

The client is a Dell 2650 running Oracle 9i

Name: Tigris

   CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Quoted 7 lines

Why are you using NFSv2? That forces the server to use synchronous writes (i.e. each write must be committed to disk before the server replies to the client) and is very inefficient.

If your application is write-intensive, you should always choose NFSv3 or greater.

There might also be some benefits from using the larger transfer sizes available in NFSv3 or greater too. NFSv2 is limited to 8kB transfers, which is kind of small for the larger bandwidth networks, especially in conjunction with the use of TCP to minimize any possible retransmitted packets.

Thanx...

ps

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
In reply to Trond Myklebust
Quoted 26 lines · Trond Myklebust
Quoted 14 lines · Stephen Carville

I have NFS server running on a Dell 2850:

Name: Amazon

   CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

Quoted 8 lines

The client is a Dell 2650 running Oracle 9i

Name: Tigris

   CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Quoted 6 lines

Why are you using NFSv2? That forces the server to use synchronous writes (i.e. each write must be committed to disk before the server replies to the client) and is very inefficient.

If your application is write-intensive, you should always choose NFSv3 or greater.

No problem. I added nfsvers=3 to the mount options and I'll see how it works tonight

Signature · 16 lines
Stephen Carville <[email protected]>
Unix and Network Admin
Nationwide Totalflood
6033 W. Century Blvd
Los Angeles, CA 90045
310-342-3602


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#4 lore raw
In reply to Stephen Carville

Did you specify using a 32k read and write size, that I didn't see somewhere? If you have a very clean network, you could get an advantage of using UDP as well. I'm not sure which transport you are using though.

Oh I see a mention of nfs v2 tcp. Well, I'd recommend moving to nfs v3, using a 64k read/write size, and using UDP. That might give you some gains. Of course, I'm usually a TCP bigot for nfs transport, but for a very clean, almost back to back, dedicated switch type of environment, I'll allow it.

-Blake

Quoted 41 lines · Stephen Carville

I have NFS server running on a Dell 2850:

Name: Amazon

   CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

The client is a Dell 2650 running Oracle 9i

Name: Tigris

   CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Both have hyperthreading enabled.
The machines are connected via a dedicated gigabit switch.

The goal (among others) is to run the nightly RMAN backups of the Oracle database to the NFS drive on Amazon where it is backed up to tape later. To this end I mount the NFS volume, run the backups, and unmount the volme.

That part works well. However the thruput on the connection is much less than I expected. Simple file transfers for example give only about:

Using NFS
Amazon to Tigris: 168 Mbps
Tigris to Amazon: 104 Mbps

Using ftp
ssh A to T: 320 Mbps
ssh T to A 360 Mbps

Quoted 26 lines

I can accept that I may never get a full Gigabit per second over copper but it still looks like there is room for improvement in the NFS trnsfer speeds.

Suggestions?

Signature · 17 lines
Stephen Carville <[email protected]>
Unix and Network Admin
Nationwide Totalflood
6033 W. Century Blvd
Los Angeles, CA 90045
310-342-3602



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw
Quoted 8 lines · Blake Golliher

Did you specify using a 32k read and write size, that I didn't see somewhere? If you have a very clean network, you could get an advantage of using UDP as well. I'm not sure which transport you are using though.

Oh I see a mention of nfs v2 tcp. Well, I'd recommend moving to nfs v3, using a 64k read/write size, and using UDP. That might give you some gains. Of course, I'm usually a TCP bigot for nfs transport, but for a very clean, almost back to back, dedicated switch type of environment, I'll allow it.

Mmm. I thought UDP was max. 8k read and write size and 32k was the max. for TCP.

Quoted 75 lines

-Blake

Quoted 41 lines · Stephen Carville

I have NFS server running on a Dell 2850:

Name: Amazon

  CPU: 3000 Mhz Xeon  (X2)

Memory: 2G
Swap: 8G
OS: Fedora Core 3
NFS: Version 1-4

I export a 150G volume thusly:

/NFS/tigris_backup 10.1.1.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check,anonuid=65534,anongid=65534)

The client is a Dell 2650 running Oracle 9i

Name: Tigris

  CPU: 3200Mhz Xeon (X2)

Memory: 8G
Swap: 4G
OS: Redhat ES 3.2
NFS: Version 2/tcp

Both have hyperthreading enabled.
The machines are connected via a dedicated gigabit switch.

The goal (among others) is to run the nightly RMAN backups of the Oracle database to the NFS drive on Amazon where it is backed up to tape later. To this end I mount the NFS volume, run the backups, and unmount the volme.

That part works well. However the thruput on the connection is much less than I expected. Simple file transfers for example give only about:

Using NFS
Amazon to Tigris: 168 Mbps
Tigris to Amazon: 104 Mbps

Using ftp
ssh A to T: 320 Mbps
ssh T to A 360 Mbps

Quoted 26 lines

I can accept that I may never get a full Gigabit per second over copper but it still looks like there is room for improvement in the NFS trnsfer speeds.

Suggestions?

Signature · 17 lines
Stephen Carville <[email protected]>
Unix and Network Admin
Nationwide Totalflood
6033 W. Century Blvd
Los Angeles, CA 90045
310-342-3602



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw
Quoted 6 lines · Ian Kent
Quoted 4 lines · Blake Golliher

Did you specify using a 32k read and write size, that I didn't see somewhere? If you have a very clean network, you could get an advantage of using UDP as well. I'm not sure which transport you are using though.

Quoted 7 lines
Quoted 4 lines

Oh I see a mention of nfs v2 tcp. Well, I'd recommend moving to nfs v3, using a 64k read/write size, and using UDP. That might give you some gains. Of course, I'm usually a TCP bigot for nfs transport, but for a very clean, almost back to back, dedicated switch type of environment, I'll allow it.

Mmm. I thought UDP was max. 8k read and write size and 32k was the max. for TCP.

The linux nfs client supports, at least, a 64k block transfer size. I don't believe it's constrained by transport protocol.

-Blake

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#7 lore raw
Quoted 15 lines · Blake Golliher
Quoted 12 lines · Ian Kent
Quoted 8 lines · Blake Golliher

Did you specify using a 32k read and write size, that I didn't see somewhere? If you have a very clean network, you could get an advantage of using UDP as well. I'm not sure which transport you are using though.

Oh I see a mention of nfs v2 tcp. Well, I'd recommend moving to nfs v3, using a 64k read/write size, and using UDP. That might give you some gains. Of course, I'm usually a TCP bigot for nfs transport, but for a very clean, almost back to back, dedicated switch type of environment, I'll allow it.

Quoted 5 lines
Quoted 2 lines

Mmm. I thought UDP was max. 8k read and write size and 32k was the max. for TCP.

The linux nfs client supports, at least, a 64k block transfer size. I don't believe it's constrained by transport protocol.

hi blake-

as of 2.6.15, the stock Linux NFS client supports up to 32KB read and write size for NFS versions 2, 3, and 4, and for UDP and TCP.

the recommended r/wsize for UDP is 8KB simply to avoid IP fragment storms. on a clean network with few other clients, 32KB with UDP will probably work well.

soon, the Linux NFS client will support up to 1MB r/wsize, depending on the network transport and the server's capabilities.

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Open Source NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA email;internet:[email protected]
title:Member of Technical Staff
tel;work:+1 734 763-4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668-1089
x-mozilla-html:FALSE
url:http://troy.citi.umich.edu/u/cel/
version:2.1
end:vcard

#8 lore raw
In reply to Blake Golliher
Quoted 2 lines · Blake Golliher

The linux nfs client supports, at least, a 64k block transfer size. I don't believe it's constrained by transport protocol.

That is a very recent feature, though. Unless you are using the NFS_ALL patches, you will only get 64k block transfer sizes on the very latest GIT tarball. ;-)

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#9 lore raw

[email protected] wrote on 01/08/2006 12:22:13 PM:

Quoted 12 lines · Blake Golliher
Quoted 3 lines

The linux nfs client supports, at least, a 64k block transfer size. I

don't believe it's constrained by transport protocol.

That is a very recent feature, though. Unless you are using the NFS_ALL patches, you will only get 64k block transfer sizes on the very latest GIT tarball. ;-)

Cheers,
Trond

But is still constrained by the transport protocol. Marc.

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#10 lore raw
Quoted 11 lines · Marc Eshel

[email protected] wrote on 01/08/2006 12:22:13 PM:

Quoted 9 lines · Blake Golliher
Quoted 3 lines

The linux nfs client supports, at least, a 64k block transfer size. I

don't believe it's constrained by transport protocol.

That is a very recent feature, though. Unless you are using the NFS_ALL patches, you will only get 64k block transfer sizes on the very latest GIT tarball. ;-)

Quoted 6 lines
Quoted 3 lines

Cheers,
Trond

But is still constrained by the transport protocol. Marc.

Yes. UDP won't allow you to use 64k read/write sizes since that would overflow the size of a datagram. TCP has no such inherent limitation, though.

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#11 lore raw

I tried lately to change the max block size on the client and server from 32K to 64K and I got 36K. Looking at the code it looked like it got the size from

max_rpc_payload = nfs_block_size(rpc_max_payload(server->client), NULL);
        if (server->rsize > max_rpc_payload)

and max_payload was set by the network driver.

Marc.

[email protected] wrote on 01/08/2006 02:09:15 PM:

Quoted 5 lines · Marc Eshel
Quoted 4 lines

[email protected] wrote on 01/08/2006 12:22:13 PM:

Quoted 2 lines · Blake Golliher
Quoted 1 line

The linux nfs client supports, at least, a 64k block transfer

size. I

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

don't believe it's constrained by transport protocol.

That is a very recent feature, though. Unless you are using the

NFS_ALL

Quoted 1 line
Quoted 1 line
Quoted 1 line

patches, you will only get 64k block transfer sizes on the very

latest

Quoted 13 lines
Quoted 8 lines
Quoted 5 lines

GIT tarball. ;-)

Cheers,
Trond

But is still constrained by the transport protocol. Marc.

Yes. UDP won't allow you to use 64k read/write sizes since that would overflow the size of a datagram. TCP has no such inherent limitation, though.

Quoted 7 lines

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log

files

Quoted 6 lines

for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs