Streaming perf problem on 10g
Subsystems (2)
Hi All,
I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.
The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.
The network is performing fine with many Gbps of iperf throughput.
Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.
I've tried changing the tcp_slot_table_entries and the wsize but there is negligible gain from these.
Does it sound like a client side inefficiency?
Thanks
-Shehjar
Quoted 6 lines · Shehjar Tikoo
Hi All,
I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.
The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.
If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.
Quoted 2 lines
The network is performing fine with many Gbps of iperf throughput.
GbE gets you 1 Gbps. 10GbE may get you from 3-10 Gbps, depending upon many things. What are your numbers?
Quoted 2 lines
Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.
Sounds like you are writing over the gigabit, and not the 10GbE interface.
Quoted 4 lines
I've tried changing the
tcp_slot_table_entriesand the wsize but there is negligible gain from these.Does it sound like a client side inefficiency?
Nope.
Signature · 8 lines
Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: [email protected] web : http://scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
Hi Shehjar,
Can you provide the exact dd command you are running both locally and for the NFS mount?
-Tommy
On Wed, Nov 3, 2010 at 11:33 AM, Joe Landman <[email protected]> wr= ote:
Quoted 5 lines · Shehjar Tikoo
Quoted 4 lines
Hi All,
I am running into a performance problem with 2.6.32-23 Ubuntu lucid =
on
Quoted 3 lines
Quoted 3 lines
both client and server.
The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb fil=
e in
Quoted 4 lines
Quoted 2 lines
64k blocks.
If the size of this file is comparable to or smaller than the client =
or
Quoted 6 lines
server ram, this number is meaningless.
Quoted 2 lines
The network is performing fine with many Gbps of iperf throughput.
GbE gets you 1 Gbps. =A010GbE may get you from 3-10 Gbps, depending u=
pon many
Quoted 3 lines
things. =A0What are =A0your numbers?
Quoted 1 line
Yet, the dd write performance over the nfs mount point ranges from 9=
6-105
Quoted 4 lines
Quoted 2 lines
Mbps for a 6gb file in 64k blocks.
Sounds like you are writing over the gigabit, and not the 10GbE inter=
face.
Quoted 2 lines
Quoted 2 lines
I've tried changing the
tcp_slot_table_entriesand the wsize but the=
re is
Quoted 7 lines
Quoted 4 lines
negligible gain from these.
Does it sound like a client side inefficiency?
Nope.
Quoted 11 lines
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman-nyOC7EYE20mM0MU9lROt9DlRY1/[email protected] web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
Quoted 3 lines
the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · [email protected] wrote:
Hi Shehjar,
Can you provide the exact dd command you are running both locally and for the NFS mount?
on the ssd:
# dd if=/dev/zero of=bigfile4 bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 0.690624 s, 1.5 GB/s # dd if=/dev/zero of=bigfile4 bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 1.72764 s, 607 MB/s
The ssd file system is ext4 mounted as (rw,noatime,nodiratime,data=writeback)
Here is another strangeness, using oflag=direct gives better performance:
On nfs mount:
# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 3.7063 s, 283 MB/s # rm /tmp/testmount/bigfile3 # dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 9.66876 s, 108 MB/s
The kernel on both server and client is 2.6.32-23, so I think this regression might be in play.
http://thread.gmane.org/gmane.comp.file-systems.ext4/20360
Thanks
-Shehjar
Quoted 47 lines
-Tommy
Quoted 43 lines · Joe Landman
Quoted 8 lines · Shehjar Tikoo
Hi All,
I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.
The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.
If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.
Quoted 2 lines
The network is performing fine with many Gbps of iperf throughput.
GbE gets you 1 Gbps. 10GbE may get you from 3-10 Gbps, depending upon many things. What are your numbers?
Quoted 3 lines
Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.
Sounds like you are writing over the gigabit, and not the 10GbE interface.
Quoted 5 lines
I've tried changing the
tcp_slot_table_entriesand the wsize but there is negligible gain from these.Does it sound like a client side inefficiency?
Nope.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: [email protected]
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Shehjar,
Have you tested with another file system besides ext4, like XFS or ReiserFS?
How many SSD's in the configuration? What is the storage controller (SAS, SATA, PCIe direct-connect)? 1.5GB/sec a lot of speed...seems like at least 8 SSDs but please confirm.
Also, you are not copying enough data in this test....how much DRAM in the server with the SSD? I would run dd with an IO amount at least double or triple the the amount of memory in the system. 1GB is not enough.
-Tommy
Quoted 98 lines · Shehjar Tikoo
Quoted 5 lines · [email protected] wrote:
Hi Shehjar,
Can you provide the exact dd command you are running both locally and for the NFS mount?
on the ssd:
# dd if=/dev/zero of=bigfile4 bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 0.690624 s, 1.5 GB/s # dd if=/dev/zero of=bigfile4 bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 1.72764 s, 607 MB/sThe ssd file system is ext4 mounted as
(rw,noatime,nodiratime,data=writeback)Here is another strangeness, using oflag=direct gives better performance:
On nfs mount:
# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 3.7063 s, 283 MB/s # rm /tmp/testmount/bigfile3 # dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 9.66876 s, 108 MB/sThe kernel on both server and client is 2.6.32-23, so I think this regression might be in play.
http://thread.gmane.org/gmane.comp.file-systems.ext4/20360
Thanks
-ShehjarQuoted 54 lines
-Tommy
Quoted 49 lines · Joe Landman
Quoted 9 lines · Shehjar Tikoo
Hi All,
I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.
The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.
If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.
Quoted 2 lines
The network is performing fine with many Gbps of iperf throughput.
GbE gets you 1 Gbps. 10GbE may get you from 3-10 Gbps, depending upon many
things. What are your numbers?Quoted 4 lines
Yet, the dd write performance over the nfs mount point ranges from 96-105
Mbps for a 6gb file in 64k blocks.Sounds like you are writing over the gigabit, and not the 10GbE interface.
Quoted 6 lines
I've tried changing the
tcp_slot_table_entriesand the wsize but there is
negligible gain from these.Does it sound like a client side inefficiency?
Nope.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: [email protected]
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html