Skip to contentSkip to conversation

Some NFS performance numbers for 2.6.31

DISCUSSION Ben Greear 2 participants 4 messages Sep 23, 2009 – Sep 24, 2009

Subsystems (2)
lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.

Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver) MTU 1500

Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)

Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunks O_DIRECT is enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.

Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.

Some things of interest:

*  Rates bounce around several Gbps
*  I see tcp retransmits in netstat -s on the server
*  I see zero errors (pkt loss, etc) reported by the NICs.
*  Reading 100M files slows down the test.
*  2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell)
*  Messing with rmem max, backlog, and other network tunings doesn't seem to matter.
*  Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd

seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx on both ports (using pktgen and 1514 byte pkts).

Here is snippet of top on the client. bthelper is the thing doing the reads:

top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98

Tasks: 227 total,   3 running, 224 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  1.4%sy,  0.0%ni, 74.4%id,  0.0%wa,  0.6%hi, 23.4%si,  0.0%st
Mem:  12326604k total,  1343680k used, 10982924k free,    30912k buffers
Swap: 14254072k total,        0k used, 14254072k free,   112084k cached
 1586 root      15  -5     0    0    0 R 39.2  0.0   1:47.91 rpciod/6
   22 root      15  -5     0    0    0 R 38.5  0.0   1:46.94 ksoftirqd/6
 3841 root       3 -17 49320  33m  980 D 20.3  0.3   0:23.05 bthelper
 3840 root       3 -17 49320  33m  980 D 18.9  0.3   0:23.43 bthelper
 3836 root       3 -17 49320  33m  980 D 12.0  0.3   0:26.39 bthelper
 3849 root       3 -17 49320  33m  980 D 11.3  0.3   0:22.74 bthelper

And, here is the server:

top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68

Tasks: 291 total,   1 running, 290 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  2.5%sy,  0.0%ni, 95.5%id,  0.0%wa,  0.2%hi,  1.7%si,  0.0%st
Mem:  12325312k total,   617720k used, 11707592k free,     8240k buffers
Swap:        0k total,        0k used,        0k free,   101016k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2171 root      15  -5     0    0    0 S  4.0  0.0   4:41.54 nfsd
 2163 root      15  -5     0    0    0 S  3.6  0.0   3:06.09 nfsd
 2166 root      15  -5     0    0    0 S  3.6  0.0   4:45.78 nfsd
 2176 root      15  -5     0    0    0 S  3.6  0.0   4:45.45 nfsd
 2164 root      15  -5     0    0    0 S  3.3  0.0   3:08.28 nfsd
 2165 root      15  -5     0    0    0 S  3.3  0.0   4:20.12 nfsd
 2167 root      15  -5     0    0    0 S  3.3  0.0   4:31.29 nfsd
 2170 root      15  -5     0    0    0 S  3.3  0.0   4:50.80 nfsd
 2174 root      15  -5     0    0    0 S  3.3  0.0   4:50.74 nfsd
 2168 root      15  -5     0    0    0 S  3.0  0.0   4:53.15 nfsd
 2169 root      15  -5     0    0    0 S  3.0  0.0   4:48.07 nfsd

I'm curious if anyone else has done similar testing...

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
author
#1 lore raw
Quoted 2 lines · Ben Greear

I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.

I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.

I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.

Take it easy,
Ben

Quoted 85 lines

Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver)
MTU 1500

Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)

Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunks O_DIRECT is enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.

Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.

Some things of interest:
* Rates bounce around several Gbps
* I see tcp retransmits in netstat -s on the server
* I see zero errors (pkt loss, etc) reported by the NICs.
* Reading 100M files slows down the test.
* 2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell)
* Messing with rmem max, backlog, and other network tunings doesn't seem to matter.
* Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd
seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max
IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx
on both ports (using pktgen and 1514 byte pkts).

Here is snippet of top on the client. bthelper is the thing doing the reads:

top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98
Tasks: 227 total, 3 running, 224 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 1.4%sy, 0.0%ni, 74.4%id, 0.0%wa, 0.6%hi, 23.4%si, 0.0%st
Mem: 12326604k total, 1343680k used, 10982924k free, 30912k buffers
Swap: 14254072k total, 0k used, 14254072k free, 112084k cached

1586 root 15 -5 0 0 0 R 39.2 0.0 1:47.91 rpciod/6 22 root 15 -5 0 0 0 R 38.5 0.0 1:46.94 ksoftirqd/6 3841 root 3 -17 49320 33m 980 D 20.3 0.3 0:23.05 bthelper 3840 root 3 -17 49320 33m 980 D 18.9 0.3 0:23.43 bthelper 3836 root 3 -17 49320 33m 980 D 12.0 0.3 0:26.39 bthelper 3849 root 3 -17 49320 33m 980 D 11.3 0.3 0:22.74 bthelper

And, here is the server:

top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68
Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.5%sy, 0.0%ni, 95.5%id, 0.0%wa, 0.2%hi, 1.7%si, 0.0%st
Mem: 12325312k total, 617720k used, 11707592k free, 8240k buffers
Swap: 0k total, 0k used, 0k free, 101016k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2171 root 15 -5 0 0 0 S 4.0 0.0 4:41.54 nfsd 2163 root 15 -5 0 0 0 S 3.6 0.0 3:06.09 nfsd 2166 root 15 -5 0 0 0 S 3.6 0.0 4:45.78 nfsd 2176 root 15 -5 0 0 0 S 3.6 0.0 4:45.45 nfsd 2164 root 15 -5 0 0 0 S 3.3 0.0 3:08.28 nfsd 2165 root 15 -5 0 0 0 S 3.3 0.0 4:20.12 nfsd 2167 root 15 -5 0 0 0 S 3.3 0.0 4:31.29 nfsd 2170 root 15 -5 0 0 0 S 3.3 0.0 4:50.80 nfsd 2174 root 15 -5 0 0 0 S 3.3 0.0 4:50.74 nfsd 2168 root 15 -5 0 0 0 S 3.0 0.0 4:53.15 nfsd 2169 root 15 -5 0 0 0 S 3.0 0.0 4:48.07 nfsd

I'm curious if anyone else has done similar testing...

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#2 lore raw
Quoted 8 lines · Ben Greear
Quoted 2 lines · Ben Greear

I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.

I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.

I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.

On a 10 gig nic? (Oh, sorry, I see: 2 10 gig nics. OK!) Just out of
curiosity: have you done any testing with real drives? Which kernel is this?

--b.

Quoted 99 lines

Take it easy,
Ben

Quoted 85 lines

Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver)
MTU 1500

Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)

Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunks O_DIRECT is enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.

Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.

Some things of interest:
* Rates bounce around several Gbps
* I see tcp retransmits in netstat -s on the server
* I see zero errors (pkt loss, etc) reported by the NICs.
* Reading 100M files slows down the test.
* 2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell)
* Messing with rmem max, backlog, and other network tunings doesn't seem to matter.
* Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd
seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max
IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx
on both ports (using pktgen and 1514 byte pkts).

Here is snippet of top on the client. bthelper is the thing doing the reads:

top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98
Tasks: 227 total, 3 running, 224 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 1.4%sy, 0.0%ni, 74.4%id, 0.0%wa, 0.6%hi, 23.4%si, 0.0%st
Mem: 12326604k total, 1343680k used, 10982924k free, 30912k buffers
Swap: 14254072k total, 0k used, 14254072k free, 112084k cached

1586 root 15 -5 0 0 0 R 39.2 0.0 1:47.91 rpciod/6 22 root 15 -5 0 0 0 R 38.5 0.0 1:46.94 ksoftirqd/6 3841 root 3 -17 49320 33m 980 D 20.3 0.3 0:23.05 bthelper 3840 root 3 -17 49320 33m 980 D 18.9 0.3 0:23.43 bthelper 3836 root 3 -17 49320 33m 980 D 12.0 0.3 0:26.39 bthelper 3849 root 3 -17 49320 33m 980 D 11.3 0.3 0:22.74 bthelper

And, here is the server:

top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68
Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.5%sy, 0.0%ni, 95.5%id, 0.0%wa, 0.2%hi, 1.7%si, 0.0%st
Mem: 12325312k total, 617720k used, 11707592k free, 8240k buffers
Swap: 0k total, 0k used, 0k free, 101016k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2171 root 15 -5 0 0 0 S 4.0 0.0 4:41.54 nfsd 2163 root 15 -5 0 0 0 S 3.6 0.0 3:06.09 nfsd 2166 root 15 -5 0 0 0 S 3.6 0.0 4:45.78 nfsd 2176 root 15 -5 0 0 0 S 3.6 0.0 4:45.45 nfsd 2164 root 15 -5 0 0 0 S 3.3 0.0 3:08.28 nfsd 2165 root 15 -5 0 0 0 S 3.3 0.0 4:20.12 nfsd 2167 root 15 -5 0 0 0 S 3.3 0.0 4:31.29 nfsd 2170 root 15 -5 0 0 0 S 3.3 0.0 4:50.80 nfsd 2174 root 15 -5 0 0 0 S 3.3 0.0 4:50.74 nfsd 2168 root 15 -5 0 0 0 S 3.0 0.0 4:53.15 nfsd 2169 root 15 -5 0 0 0 S 3.0 0.0 4:48.07 nfsd

I'm curious if anyone else has done similar testing...

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#3 lore raw
Quoted 5 lines · J. Bruce Fields
Quoted 4 lines · Ben Greear
Quoted 2 lines · Ben Greear

I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.

Quoted 8 lines
Quoted 4 lines

I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.

I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.

On a 10 gig nic? (Oh, sorry, I see: 2 10 gig nics. OK!) Just out of
curiosity: have you done any testing with real drives? Which kernel is this?

I see similar performance on real drivers, but I haven't done any tests with *only* using physical drivers. Since that would exercise only a single mount, I'm not sure if it would scale as well as using multiple mounts with multiple virtual interfaces, but that is just pure hypothesizing on my part.

This is 2.6.31 kernel, 64-bit, most kernel related debugging disabled.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com