Slow NFS loop performance.
Subsystems (2)
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 20 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
Any ideas why?
My server configuration (NFS4) -- anonuid=1000,anongid=1000,**secure,no_subtree_check,rw,**fsid=0,crossmnt
My client mount options -- rsize=1048576,wsize=1048576,**async,hard,timeo=60,retrans=2,** actimeo=900,retry=2,**lookupcache=pos,nfsvers=4,** nolock,intr,noacl,rdirplus
Quoted 14 lines · dE .
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 9 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
What exactly is your test?
--b.
Quoted 16 lines
Quoted 11 lines
Any ideas why?
My server configuration (NFS4) -- anonuid=1000,anongid=1000,**secure,no_subtree_check,rw,**fsid=0,crossmnt
My client mount options -- rsize=1048576,wsize=1048576,**async,hard,timeo=60,retrans=2,** actimeo=900,retry=2,**lookupcache=pos,nfsvers=4,** nolock,intr,noacl,rdirplus
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
You're using a non-default timeo=60 on a TCP connection.
Why?
Quoted 39 lines
-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of dE .
Sent: Friday, November 23, 2012 6:02 AM
To: [email protected]
Subject: Slow NFS loop performance.Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 5 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the >
write performance of the client on the loop mounted share is pretty bad.
Quoted 2 lines
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps >
speeds, whereas on the loop mounted device, I get at best 6MBps.
Quoted 5 lines
Any ideas why?
My server configuration (NFS4) --
anonuid=1000,anongid=1000,**secure,no_subtree_check,rw,**fsid=0,cross mnt
Quoted 6 lines
My client mount options -- rsize=1048576,wsize=1048576,**async,hard,timeo=60,retrans=2,** actimeo=900,retry=2,**lookupcache=pos,nfsvers=4,** nolock,intr,noacl,rdirplus
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · J. Bruce Fields
Quoted 9 lines · dE .
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 4 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
What exactly is your test?
--b.
Sorry for the late response. I'd 200+ unread mails.
I'm writing a large file to the mounted loop device.
Quoted 3 lines · Trond
You're using a non-default timeo=60 on a TCP connection.
Why?
This's a LAN with just 2 PCs. So there're no packet drops, so there's no point resending requests frequently.
Quoted 16 lines · dE .
Quoted 15 lines · J. Bruce Fields
Quoted 14 lines · dE .
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 9 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
Quoted 7 lines
Quoted 3 lines
What exactly is your test?
--b.
Sorry for the late response. I'd 200+ unread mails.
I'm writing a large file to the mounted loop device.
How large, and do you have the exact command you're using for that?
Also, what are the client and server versions?
I don't have any good idea off the top of my head. I doubt anyone's worked on optimizing exports of loop devices. I guess the first step would be to collect some statistics in the two cases (loop device and non-loop device), compare them, and see if you can see any patterns. /proc/self/mountstats on the client, and /proc/fs/nfsd/pool_stats, on the server, would be starting points. Maybe perf on the server could also show up something. Just running "top" on the server might be interesting. (E.g. is the CPU obviously busier in the slow case?)
--b.
Quoted 36 lines · J. Bruce Fields
Quoted 14 lines · dE .
Quoted 13 lines · J. Bruce Fields
Quoted 12 lines · dE .
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 7 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
What exactly is your test?
--b.
Sorry for the late response. I'd 200+ unread mails.
I'm writing a large file to the mounted loop device.
How large, and do you have the exact command you're using for that?
Also, what are the client and server versions?
I don't have any good idea off the top of my head. I doubt anyone's worked on optimizing exports of loop devices. I guess the first step would be to collect some statistics in the two cases (loop device and non-loop device), compare them, and see if you can see any patterns.
/proc/self/mountstatson the client, and/proc/fs/nfsd/pool_stats, on the server, would be starting points. Maybe perf on the server could also show up something. Just running "top" on the server might be interesting. (E.g. is the CPU obviously busier in the slow case?)
Quoted 1 line
--b.
I'll try it out, but I've see no CPU usage problem here. This time I'll use dd (I did that previously also).
Quoted 7 lines
-----Original Message-----
From: [email protected] [mailto:linux-nfs- [email protected]] On Behalf Of dE .
Sent: Monday, November 26, 2012 11:01 AM
To: [email protected]
Subject: Re: Slow NFS loop performance.
Quoted 7 lines
Quoted 3 lines · Trond
You're using a non-default timeo=60 on a TCP connection.
Why?
This's a LAN with just 2 PCs. So there're no packet drops, so there's no point resending requests frequently.
The default is timeo=600.
Timeo=60 is a 6 second timeout, and is not only pointless for a TCP connection, it can actually cause unnecessary disruptions when you use NFSv4 because all retransmissions are required to do a TCP disconnect+reconnect cycle.
Trond
Quoted 36 lines · J. Bruce Fields
Quoted 14 lines · dE .
Quoted 13 lines · J. Bruce Fields
Quoted 12 lines · dE .
Humm... The last thing I expected was no response even in the mailing list.
So I'm filing a bug on this.
Quoted 7 lines · dE .
Hi!
Great job with NFS server, it surely is fast, but not on loop devices.
If I loop mount a file and share the mount point over NFS3 or NFS4, the write performance of the client on the loop mounted share is pretty bad.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps speeds, whereas on the loop mounted device, I get at best 6MBps.
What exactly is your test?
--b.
Sorry for the late response. I'd 200+ unread mails.
I'm writing a large file to the mounted loop device.
How large, and do you have the exact command you're using for that?
Also, what are the client and server versions?
I don't have any good idea off the top of my head. I doubt anyone's worked on optimizing exports of loop devices. I guess the first step would be to collect some statistics in the two cases (loop device and non-loop device), compare them, and see if you can see any patterns.
/proc/self/mountstatson the client, and/proc/fs/nfsd/pool_stats, on the server, would be starting points. Maybe perf on the server could also show up something. Just running "top" on the server might be interesting. (E.g. is the CPU obviously busier in the slow case?)
Quoted 1 line
--b.
It's clear --
On the loop device --
dd if=/dev/zero of=/mnt/shares/mount_point/xfs_large_files/test.zero bs=1M count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1788.51 s, 600 kB/s
And otherwise --
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 157.447 s, 6.8 MB/s
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 128.628 s, 8.3 MB/s
I ran out of patience to do any tests on the loop device.
cat /proc/self/mountstats --
device rootfs mounted on / with fstype rootfs
device /dev/root mounted on / with fstype reiserfs
..
..
device 192.168.1.3:/ mounted on /mnt/shares with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 4026
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 14 54 0 8523 6 5 47 611072 0 0 611072 3040 12 6 12 12 0
6 0 5 611072 0 0 0 0 0 0
bytes: 0 2502950912 0 0 0 2502950912 0 611072
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 10035 10035 0 2505279032 1324620 53001291 3776691
56778324
COMMIT: 48 48 0 10176 5952 151346 14030 165378
OPEN: 6 6 0 1784 2352 0 43 43
OPEN_CONFIRM: 2 2 0 408 136 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 6 6 0 1320 792 0 1 1
SETATTR: 6 6 0 1432 1440 0 3478 3479
FSINFO: 2 2 0 336 216 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 13 13 0 2568 1488 15581 8484 28896
GETATTR: 15 15 0 2860 3300 0 7 8
LOOKUP: 6 7 0 1472 516 19755 11318 31074
LOOKUP_ROOT: 1 1 0 160 240 0 26 27
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 164 72 0 0 0
STATFS: 6 6 0 1128 696 0 1 1
READLINK: 0 0 0 0 0 0 0 0
READDIR: 3 3 0 652 3380 0 42 42
SERVER_CAPS: 3 3 0 492 276 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0device 192.168.1.3:/mount_point/xfs_large_files/ mounted on /mnt/shares/mount_point/xfs_large_files with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,port=0,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 2344
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 0 0 0 6464 0 0 2 262144 0 0 262144 10131 0 1 2 2 0 1 0
0 262144 0 0 0 0 0 0
bytes: 0 1073741824 0 0 0 1073741824 0 262144
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 7821 12504 5 1360145244 1032372 461231706
30244294 493689738
COMMIT: 23 28 0 5600 2852 981824 61062 1049646
OPEN: 1 1 0 336 404 0 122 122
OPEN_CONFIRM: 1 1 0 216 68 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 1 1 0 232 132 0 0 0
SETATTR: 1 1 0 244 240 0 0 0
FSINFO: 1 1 0 200 108 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 1 1 0 208 124 0 0 0
GETATTR: 1 1 0 200 220 0 0 0
LOOKUP: 0 0 0 0 0 0 0 0
LOOKUP_ROOT: 0 0 0 0 0 0 0 0
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 196 72 0 0 0
STATFS: 0 0 0 0 0 0 0 0
READLINK: 0 0 0 0 0 0 0 0
READDIR: 0 0 0 0 0 0 0 0
SERVER_CAPS: 2 2 0 392 184 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0And /proc/fs/nfsd/pool_stats
cat /proc/fs/nfsd/pool_stats
# pool packets-arrived sockets-enqueued threads-woken overloads-avoided threads-timedout 0 4637559 21028 1689208 269 0
Server --
uname -r
2.6.32-5-amd64
modinfo nfs
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfs/nfs.ko license: GPL author: Olaf Kirch <[email protected]> depends: sunrpc,fscache,lockd,auth_rpcgss,nfs_acl vermagic: 2.6.32-5-amd64 SMP mod_unload modversions parm: callback_tcpport:portnr parm: cache_getent:Path to the client cache upcall program (string) parm: cache_getent_timeout:Timeout (in seconds) after which the cache upcall is assumed to have failed (ulong) parm: enable_ino64:bool
modinfo nfsd
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfsd/nfsd.ko license: GPL author: Olaf Kirch <[email protected]> depends: auth_rpcgss,sunrpc,lockd,exportfs,nfs_acl vermagic: 2.6.32-5-amd64 SMP mod_unload modversions
Client --
uname -r
3.4.2-gentoo-r1
modinfo nfs
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfs/nfs.ko license: GPL author: Olaf Kirch <[email protected]> depends: sunrpc,lockd,auth_rpcgss,nfs_acl intree: Y vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions parm: callback_tcpport:portnr parm: nfs_idmap_cache_timeout:int parm: send_implementation_id:Send implementation ID with NFSv4.1 exchange_id (ushort) parm: max_session_slots:Maximum number of outstanding NFSv4.1 requests the client will negotiate (ushort) parm: cache_getent:Path to the client cache upcall program (string) parm: cache_getent_timeout:Timeout (in seconds) after which the cache upcall is assumed to have failed (ulong) parm: enable_ino64:bool parm: nfs4_disable_idmapping:Turn off NFSv4 idmapping when using 'sec=sys' (bool)
modinfo nfsd
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfsd/nfsd.ko license: GPL author: Olaf Kirch <[email protected]> depends: auth_rpcgss,sunrpc,lockd,nfs_acl intree: Y vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions parm: nfs4_disable_idmapping:Turn off server's NFSv4 idmapping when using 'sec=sys' (bool)
net-fs/nfs-utils version 1.2.3-r1
This problem was reproducible from day 1, even during the 2.6 days. This also happen(ed/s) when I loop mount some file (on the server) in the client.
I also had an old Debian testing system which (Wheezy) which had 3.0 kernel and shared the same problem.