Skip to contentSkip to conversation

Streaming perf problem on 10g

DISCUSSION Shehjar Tikoo 3 participants 5 messages Nov 3, 2010 – Nov 5, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi All,

I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.

The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.

The network is performing fine with many Gbps of iperf throughput.

Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.

I've tried changing the tcp_slot_table_entries and the wsize but there is negligible gain from these.

Does it sound like a client side inefficiency?

Thanks
-Shehjar

#1 lore raw
Quoted 6 lines · Shehjar Tikoo

Hi All,

I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.

The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.

If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.

Quoted 2 lines

The network is performing fine with many Gbps of iperf throughput.

GbE gets you 1 Gbps. 10GbE may get you from 3-10 Gbps, depending upon many things. What are your numbers?

Quoted 2 lines

Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.

Sounds like you are writing over the gigabit, and not the 10GbE interface.

Quoted 4 lines

I've tried changing the tcp_slot_table_entries and the wsize but there is negligible gain from these.

Does it sound like a client side inefficiency?

Nope.

Signature · 8 lines
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: [email protected]
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
#2 lore raw

Hi Shehjar,

Can you provide the exact dd command you are running both locally and for the NFS mount?

-Tommy

On Wed, Nov 3, 2010 at 11:33 AM, Joe Landman <[email protected]> wr= ote:

Quoted 5 lines · Shehjar Tikoo
Quoted 4 lines

Hi All,

I am running into a performance problem with 2.6.32-23 Ubuntu lucid =

on

Quoted 3 lines
Quoted 3 lines

both client and server.

The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb fil=

e in

Quoted 4 lines
Quoted 2 lines

64k blocks.

If the size of this file is comparable to or smaller than the client =

or

Quoted 6 lines

server ram, this number is meaningless.

Quoted 2 lines

The network is performing fine with many Gbps of iperf throughput.

GbE gets you 1 Gbps. =A010GbE may get you from 3-10 Gbps, depending u=

pon many

Quoted 3 lines

things. =A0What are =A0your numbers?

Quoted 1 line

Yet, the dd write performance over the nfs mount point ranges from 9=

6-105

Quoted 4 lines
Quoted 2 lines

Mbps for a 6gb file in 64k blocks.

Sounds like you are writing over the gigabit, and not the 10GbE inter=

face.

Quoted 2 lines
Quoted 2 lines

I've tried changing the tcp_slot_table_entries and the wsize but the=

re is

Quoted 7 lines
Quoted 4 lines

negligible gain from these.

Does it sound like a client side inefficiency?

Nope.

Quoted 11 lines

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman-nyOC7EYE20mM0MU9lROt9DlRY1/[email protected] web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" =

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 4 lines · [email protected] wrote:

Hi Shehjar,

Can you provide the exact dd command you are running both locally and for the NFS mount?

on the ssd:

# dd if=/dev/zero of=bigfile4 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.690624 s, 1.5 GB/s
# dd if=/dev/zero of=bigfile4 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.72764 s, 607 MB/s

The ssd file system is ext4 mounted as (rw,noatime,nodiratime,data=writeback)

Here is another strangeness, using oflag=direct gives better performance:

On nfs mount:

# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.7063 s, 283 MB/s
# rm /tmp/testmount/bigfile3
# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.66876 s, 108 MB/s

The kernel on both server and client is 2.6.32-23, so I think this regression might be in play.

http://thread.gmane.org/gmane.comp.file-systems.ext4/20360

Thanks
-Shehjar

Quoted 47 lines

-Tommy

Quoted 43 lines · Joe Landman
Quoted 8 lines · Shehjar Tikoo

Hi All,

I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.

The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.

If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.

Quoted 2 lines

The network is performing fine with many Gbps of iperf throughput.

GbE gets you 1 Gbps. 10GbE may get you from 3-10 Gbps, depending upon many things. What are your numbers?

Quoted 3 lines

Yet, the dd write performance over the nfs mount point ranges from 96-105 Mbps for a 6gb file in 64k blocks.

Sounds like you are writing over the gigabit, and not the 10GbE interface.

Quoted 5 lines

I've tried changing the tcp_slot_table_entries and the wsize but there is negligible gain from these.

Does it sound like a client side inefficiency?

Nope.

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: [email protected]
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

Hi Shehjar,

Have you tested with another file system besides ext4, like XFS or ReiserFS?

How many SSD's in the configuration? What is the storage controller (SAS, SATA, PCIe direct-connect)? 1.5GB/sec a lot of speed...seems like at least 8 SSDs but please confirm.

Also, you are not copying enough data in this test....how much DRAM in the server with the SSD? I would run dd with an IO amount at least double or triple the the amount of memory in the system. 1GB is not enough.

-Tommy

Quoted 98 lines · Shehjar Tikoo
Quoted 5 lines · [email protected] wrote:

Hi Shehjar,

Can you provide the exact dd command you are running both locally and for the NFS mount?

on the ssd:

# dd if=/dev/zero of=bigfile4 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.690624 s, 1.5 GB/s
# dd if=/dev/zero of=bigfile4 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.72764 s, 607 MB/s

The ssd file system is ext4 mounted as
(rw,noatime,nodiratime,data=writeback)

Here is another strangeness, using oflag=direct gives better performance:

On nfs mount:

# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.7063 s, 283 MB/s
# rm /tmp/testmount/bigfile3
# dd if=/dev/zero of=/tmp/testmount/bigfile3 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.66876 s, 108 MB/s

The kernel on both server and client is 2.6.32-23, so I think this regression might be in play.

http://thread.gmane.org/gmane.comp.file-systems.ext4/20360

Thanks
-Shehjar

Quoted 54 lines

-Tommy

Quoted 49 lines · Joe Landman
Quoted 9 lines · Shehjar Tikoo

Hi All,

I am running into a performance problem with 2.6.32-23 Ubuntu lucid on both client and server.

The disk is an SSD performing at 1.4 - 1.6Gbps for a dd of a 6gb file in 64k blocks.

If the size of this file is comparable to or smaller than the client or server ram, this number is meaningless.

Quoted 2 lines

The network is performing fine with many Gbps of iperf throughput.

GbE gets you 1 Gbps.  10GbE may get you from 3-10 Gbps, depending upon many
things.  What are  your numbers?

Quoted 4 lines

Yet, the dd write performance over the nfs mount point ranges from 96-105
Mbps for a 6gb file in 64k blocks.

Sounds like you are writing over the gigabit, and not the 10GbE interface.

Quoted 6 lines

I've tried changing the tcp_slot_table_entries and the wsize but there is
negligible gain from these.

Does it sound like a client side inefficiency?

Nope.

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: [email protected]
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html