Status update on Sun Neptune 10Gbit fibre using the NIU-driver.
Subsystems (2)
Hi.
Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles getting both performance and stabillity out of a Sun Neptune 10Gbit NIC over fibre. The NIU driver blew up on the system under load but Matheos Worku send me an internal Sun driver (nxge) that performed fairly well. It peaked out around 800-850 MB/s .. it put a fairly high load on the host system and peaked with over 300.000 cs/s (both numbers measured
with dstat).
Today I got around to tesing the NIU(2.6.27.20) driver again. Having repeated the test I did last summer I couldn't get it to "blow up". But instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s now and the load on the host is nearly invisible (<2) .. cs rates less than 10K mostly.
I'm not using any kind of Jumbo frames in the setup.
I'll keep it up on the niu-driver for now and report back if it encounters any problems.
In the test .. I do dd over NFS, default exports, default mount options .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS server). I have tried to put cpu-load in the NFS-server while running the NFS-server and pushing some data around on them memory subsystem while doing it.
This is just excellent.. (crossing fingers that it can beat the 180 days of uptime the nxge-driver got).
Link to old struggles:
http://thread.gmane.org/gmane.linux.kernel/677545
Jesper
Signature · 1 line
Jesper
Quoted 27 lines · Jesper Krogh
Hi.
Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles getting both performance and stabillity out of a Sun Neptune 10Gbit NIC over fibre. The NIU driver blew up on the system under load but Matheos Worku send me an internal Sun driver (nxge) that performed fairly well. It peaked out around 800-850 MB/s .. it put a fairly high load on the host system and peaked with over 300.000 cs/s (both numbers measured with dstat).
Today I got around to tesing the NIU(2.6.27.20) driver again. Having repeated the test I did last summer I couldn't get it to "blow up". But instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s now and the load on the host is nearly invisible (<2) .. cs rates less than 10K mostly.
I'm not using any kind of Jumbo frames in the setup.
I'll keep it up on the niu-driver for now and report back if it encounters any problems.
In the test .. I do dd over NFS, default exports, default mount options .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS server). I have tried to put cpu-load in the NFS-server while running the NFS-server and pushing some data around on them memory subsystem while doing it.
Quoted 7 lines
This is just excellent.. (crossing fingers that it can beat the 180 days of uptime the nxge-driver got).
Link to old struggles:
http://thread.gmane.org/gmane.linux.kernel/677545Jesper
Yes I remember this stuff. David added multi tx queue support in last July (for 2.6.27)
Could you post more information please, about load on individual cpus
for example ? (top snapshot with one line per cpu)
Is NFS using TCP or UDP on your setup ?
An interesting test would be the reverse path (transmit from clients to this server)
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Jesper Krogh
Hi.
Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles getting both performance and stabillity out of a Sun Neptune 10Gbit NIC over fibre. The NIU driver blew up on the system under load but Matheos Worku send me an internal Sun driver (nxge) that performed fairly well. It peaked out around 800-850 MB/s .. it put a fairly high load on the host system and peaked with over 300.000 cs/s (both numbers measured with dstat).
Today I got around to tesing the NIU(2.6.27.20) driver again. Having repeated the test I did last summer I couldn't get it to "blow up". But instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s now and the load on the host is nearly invisible (<2) .. cs rates less than 10K mostly.
I'm not using any kind of Jumbo frames in the setup.
I'll keep it up on the niu-driver for now and report back if it encounters any problems.
In the test .. I do dd over NFS, default exports, default mount options .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS server). I have tried to put cpu-load in the NFS-server while running the NFS-server and pushing some data around on them memory subsystem while doing it.
This is just excellent.. (crossing fingers that it can beat the 180 days of uptime the nxge-driver got).
Quoted 7 lines
Quoted 4 lines
Link to old struggles:
http://thread.gmane.org/gmane.linux.kernel/677545Jesper
Yes I remember this stuff. David added multi tx queue support in last July (for 2.6.27)
Can there be something about "fairness" changed? While producing the screenshot to you below I got this one on the server:do_ypcall: clnt_call: RPC: Unable to send; errno = No buffer space available
So NIS wasn't able to get throgh..
Quoted 2 lines
Could you post more information please, about load on individual cpus
for example ? (top snapshot with one line per cpu)
http://krogh.cc/~jesper/top.png
Quoted 1 line
Is NFS using TCP or UDP on your setup ?
TCP.
Quoted 2 lines
An interesting test would be the reverse path (transmit from clients to this server)
Thats harder to setup and not be "disk IO-bound on the server". Perhaps exporting a RAM-disk and write into that?
Quoted 2 lines
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
I'll put that on my todo list..
Signature · 1 line
Jesper
Quoted 2 lines · Eric Dumazet
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.
Jesper
Signature · 1 line
Jesper
Quoted 5 lines · Jesper Krogh
Quoted 2 lines · Eric Dumazet
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.
Am I wrong thinking that theoretical max should be around 1150-1200MB/s ?
Signature · 1 line
Jesper
Quoted 10 lines · Jesper Krogh
Quoted 6 lines · Jesper Krogh
Quoted 2 lines · Eric Dumazet
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.
Am I wrong thinking that theoretical max should be around 1150-1200MB/s ?
I dont know, what would be the current bottleneck ?
Quoted 7 lines · Jesper Krogh
Quoted 2 lines · Eric Dumazet
Also, testing 2.6.29 could be interesting, since UDP receive path doesnt need to use a global rwlock anymore.
... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.
Jesper
Wow ! Nice to ear !