Some NFS performance numbers for 2.6.31
Subsystems (2)
I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.
Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver) MTU 1500
Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)
Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunks O_DIRECT is enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.
Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.
Some things of interest:
* Rates bounce around several Gbps * I see tcp retransmits in netstat -s on the server * I see zero errors (pkt loss, etc) reported by the NICs. * Reading 100M files slows down the test. * 2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell) * Messing with rmem max, backlog, and other network tunings doesn't seem to matter. * Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd
seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx on both ports (using pktgen and 1514 byte pkts).
Here is snippet of top on the client. bthelper is the thing doing the reads:
top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98
Tasks: 227 total, 3 running, 224 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 1.4%sy, 0.0%ni, 74.4%id, 0.0%wa, 0.6%hi, 23.4%si, 0.0%st Mem: 12326604k total, 1343680k used, 10982924k free, 30912k buffers Swap: 14254072k total, 0k used, 14254072k free, 112084k cached
1586 root 15 -5 0 0 0 R 39.2 0.0 1:47.91 rpciod/6 22 root 15 -5 0 0 0 R 38.5 0.0 1:46.94 ksoftirqd/6 3841 root 3 -17 49320 33m 980 D 20.3 0.3 0:23.05 bthelper 3840 root 3 -17 49320 33m 980 D 18.9 0.3 0:23.43 bthelper 3836 root 3 -17 49320 33m 980 D 12.0 0.3 0:26.39 bthelper 3849 root 3 -17 49320 33m 980 D 11.3 0.3 0:22.74 bthelper
And, here is the server:
top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68
Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.5%sy, 0.0%ni, 95.5%id, 0.0%wa, 0.2%hi, 1.7%si, 0.0%st Mem: 12325312k total, 617720k used, 11707592k free, 8240k buffers Swap: 0k total, 0k used, 0k free, 101016k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2171 root 15 -5 0 0 0 S 4.0 0.0 4:41.54 nfsd 2163 root 15 -5 0 0 0 S 3.6 0.0 3:06.09 nfsd 2166 root 15 -5 0 0 0 S 3.6 0.0 4:45.78 nfsd 2176 root 15 -5 0 0 0 S 3.6 0.0 4:45.45 nfsd 2164 root 15 -5 0 0 0 S 3.3 0.0 3:08.28 nfsd 2165 root 15 -5 0 0 0 S 3.3 0.0 4:20.12 nfsd 2167 root 15 -5 0 0 0 S 3.3 0.0 4:31.29 nfsd 2170 root 15 -5 0 0 0 S 3.3 0.0 4:50.80 nfsd 2174 root 15 -5 0 0 0 S 3.3 0.0 4:50.74 nfsd 2168 root 15 -5 0 0 0 S 3.0 0.0 4:53.15 nfsd 2169 root 15 -5 0 0 0 S 3.0 0.0 4:48.07 nfsd
I'm curious if anyone else has done similar testing...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 2 lines · Ben Greear
I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.
I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.
I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.
Take it easy,
Ben
Quoted 85 lines
Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver)
MTU 1500Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunksO_DIRECTis enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.Some things of interest:
* Rates bounce around several Gbps
* I see tcp retransmits in netstat -s on the server
* I see zero errors (pkt loss, etc) reported by the NICs.
* Reading 100M files slows down the test.
* 2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell)
* Messing with rmem max, backlog, and other network tunings doesn't seem to matter.
* Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd
seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max
IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx
on both ports (using pktgen and 1514 byte pkts).Here is snippet of top on the client. bthelper is the thing doing the reads:
top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98
Tasks: 227 total, 3 running, 224 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 1.4%sy, 0.0%ni, 74.4%id, 0.0%wa, 0.6%hi, 23.4%si, 0.0%st
Mem: 12326604k total, 1343680k used, 10982924k free, 30912k buffers
Swap: 14254072k total, 0k used, 14254072k free, 112084k cached1586 root 15 -5 0 0 0 R 39.2 0.0 1:47.91 rpciod/6 22 root 15 -5 0 0 0 R 38.5 0.0 1:46.94 ksoftirqd/6 3841 root 3 -17 49320 33m 980 D 20.3 0.3 0:23.05 bthelper 3840 root 3 -17 49320 33m 980 D 18.9 0.3 0:23.43 bthelper 3836 root 3 -17 49320 33m 980 D 12.0 0.3 0:26.39 bthelper 3849 root 3 -17 49320 33m 980 D 11.3 0.3 0:22.74 bthelper
And, here is the server:
top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68
Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.5%sy, 0.0%ni, 95.5%id, 0.0%wa, 0.2%hi, 1.7%si, 0.0%st
Mem: 12325312k total, 617720k used, 11707592k free, 8240k buffers
Swap: 0k total, 0k used, 0k free, 101016k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2171 root 15 -5 0 0 0 S 4.0 0.0 4:41.54 nfsd 2163 root 15 -5 0 0 0 S 3.6 0.0 3:06.09 nfsd 2166 root 15 -5 0 0 0 S 3.6 0.0 4:45.78 nfsd 2176 root 15 -5 0 0 0 S 3.6 0.0 4:45.45 nfsd 2164 root 15 -5 0 0 0 S 3.3 0.0 3:08.28 nfsd 2165 root 15 -5 0 0 0 S 3.3 0.0 4:20.12 nfsd 2167 root 15 -5 0 0 0 S 3.3 0.0 4:31.29 nfsd 2170 root 15 -5 0 0 0 S 3.3 0.0 4:50.80 nfsd 2174 root 15 -5 0 0 0 S 3.3 0.0 4:50.74 nfsd 2168 root 15 -5 0 0 0 S 3.0 0.0 4:53.15 nfsd 2169 root 15 -5 0 0 0 S 3.0 0.0 4:48.07 nfsd
I'm curious if anyone else has done similar testing...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 8 lines · Ben Greear
Quoted 2 lines · Ben Greear
I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.
I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.
I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.
On a 10 gig nic? (Oh, sorry, I see: 2 10 gig nics. OK!) Just out of
curiosity: have you done any testing with real drives? Which kernel is this?
--b.
Quoted 99 lines
Take it easy,
BenQuoted 85 lines
Both: 64-bit linux.
Dual-port 10G NIC, 82599 (I think, at any rate it's the new Intel 5GT/s pci-e gen2 chipset, ixgbe driver)
MTU 1500Server: dual Intel E5530 2.4Ghz processors.
Serving 2 25M files from tmpfs (RAM)Client: Core i7 3.2Ghz, quad-core.
I'm running 10 mac-vlans on each physical interface, one NFS mount per interface
(my patches are to allow multiple mounts per client OS).
one reader for each mac-vlan and one on the physical, reading 16m chunksO_DIRECTis enabled for the readers.
Mounts are using 'noatime', leaving everything else at defaults.Total read bandwidth is about 12Gbps, but it varies quite a bit and I've seen short term (10 seconds or so)
averages go up to 15Gbps. These are on-the-wire stats reported by the NICs, not actual NFS throughput.Some things of interest:
* Rates bounce around several Gbps
* I see tcp retransmits in netstat -s on the server
* I see zero errors (pkt loss, etc) reported by the NICs.
* Reading 100M files slows down the test.
* 2M and 16M reads are about equivalent (the normal bouncing of the rates makes it hard to tell)
* Messing with rmem max, backlog, and other network tunings doesn't seem to matter.
* Running 10 readers (5 on each physical NIC) ran at 16Gbps for a few seconds (higher than I'd
seen with 10..but then it went back down to around 13Gbps.
* Running 6 seems slower..about 11.5Gbps on average.
* Using 9000 MTU yields a fairly steady 16.5Gbps throughput. This may be about the max
IO bandwidth for the server machine...but I know the client can do full 10Gbps tx + rx
on both ports (using pktgen and 1514 byte pkts).Here is snippet of top on the client. bthelper is the thing doing the reads:
top - 17:26:25 up 13 min, 3 users, load average: 25.45, 23.12, 13.98
Tasks: 227 total, 3 running, 224 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 1.4%sy, 0.0%ni, 74.4%id, 0.0%wa, 0.6%hi, 23.4%si, 0.0%st
Mem: 12326604k total, 1343680k used, 10982924k free, 30912k buffers
Swap: 14254072k total, 0k used, 14254072k free, 112084k cached1586 root 15 -5 0 0 0 R 39.2 0.0 1:47.91 rpciod/6 22 root 15 -5 0 0 0 R 38.5 0.0 1:46.94 ksoftirqd/6 3841 root 3 -17 49320 33m 980 D 20.3 0.3 0:23.05 bthelper 3840 root 3 -17 49320 33m 980 D 18.9 0.3 0:23.43 bthelper 3836 root 3 -17 49320 33m 980 D 12.0 0.3 0:26.39 bthelper 3849 root 3 -17 49320 33m 980 D 11.3 0.3 0:22.74 bthelper
And, here is the server:
top - 17:28:02 up 2:03, 2 users, load average: 0.08, 0.47, 0.68
Tasks: 291 total, 1 running, 290 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 2.5%sy, 0.0%ni, 95.5%id, 0.0%wa, 0.2%hi, 1.7%si, 0.0%st
Mem: 12325312k total, 617720k used, 11707592k free, 8240k buffers
Swap: 0k total, 0k used, 0k free, 101016k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2171 root 15 -5 0 0 0 S 4.0 0.0 4:41.54 nfsd 2163 root 15 -5 0 0 0 S 3.6 0.0 3:06.09 nfsd 2166 root 15 -5 0 0 0 S 3.6 0.0 4:45.78 nfsd 2176 root 15 -5 0 0 0 S 3.6 0.0 4:45.45 nfsd 2164 root 15 -5 0 0 0 S 3.3 0.0 3:08.28 nfsd 2165 root 15 -5 0 0 0 S 3.3 0.0 4:20.12 nfsd 2167 root 15 -5 0 0 0 S 3.3 0.0 4:31.29 nfsd 2170 root 15 -5 0 0 0 S 3.3 0.0 4:50.80 nfsd 2174 root 15 -5 0 0 0 S 3.3 0.0 4:50.74 nfsd 2168 root 15 -5 0 0 0 S 3.0 0.0 4:53.15 nfsd 2169 root 15 -5 0 0 0 S 3.0 0.0 4:48.07 nfsd
I'm curious if anyone else has done similar testing...
Thanks,
BenSignature · 7 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · J. Bruce Fields
Quoted 4 lines · Ben Greear
Quoted 2 lines · Ben Greear
I'm running some performance tests on NFSv3 on a slightly hacked 2.6.31 kernel.
Quoted 8 lines
Quoted 4 lines
I realized that LRO on the NICs was disabled because I had enabled ip-forwarding.
I re-enabled that, and now can get about 18Gbps read rates (on the wires), using MTU 1500.
On a 10 gig nic? (Oh, sorry, I see: 2 10 gig nics. OK!) Just out of
curiosity: have you done any testing with real drives? Which kernel is this?
I see similar performance on real drivers, but I haven't done any tests with *only* using physical drivers. Since that would exercise only a single mount, I'm not sure if it would scale as well as using multiple mounts with multiple virtual interfaces, but that is just pure hypothesizing on my part.
This is 2.6.31 kernel, 64-bit, most kernel related debugging disabled.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com