Skip to contentSkip to conversation

How to monitor Linux NFS client load?

DISCUSSION Andrey Borzenkov 2 participants 5 messages Jul 11, 2009 – Jul 12, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Recently we have the case of very high latencies on NFS reads as reported by application (SAP R/3). NFS server was NetApp FAS; according to NetApp statistic, average volume read latencies were in order 10ms, while SAP stats gave 30-50ms. Systems were interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.

On advice of my colleague we changed sunrpc.tcp_slot_table_entries from default 16 to 128 which seemed to make situation much better - without changing load pattern of filer in any visible way.

Now, I can understand, why we observed much higher latency on system and why changing (what effectively is) queue depth helped. But I am totally frustrated that there does not appear to be *any* possibility to detect this situation on Linux side and to get a real numbers of real NFS IO latencies or number of requests waiting to be executed (and I do not even dream about per-mount point stats).

I am grateful for any hints how can we monitor Linux NFS client and get real-life numbers of what happens inside. Thank you!

-andrey

#1 lore raw
Quoted 17 lines · Andrey Borzenkov

Recently we have the case of very high latencies on NFS reads as reported by application (SAP R/3). NFS server was NetApp FAS; according to NetApp statistic, average volume read latencies were in order 10ms, while SAP stats gave 30-50ms. Systems were interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.

On advice of my colleague we changed sunrpc.tcp_slot_table_entries from default 16 to 128 which seemed to make situation much better - without changing load pattern of filer in any visible way.

Now, I can understand, why we observed much higher latency on system and why changing (what effectively is) queue depth helped. But I am totally frustrated that there does not appear to be *any* possibility to detect this situation on Linux side and to get a real numbers of real NFS IO latencies or number of requests waiting to be executed (and I do not even dream about per-mount point stats).

Quoted 2 lines

I am grateful for any hints how can we monitor Linux NFS client and get real-life numbers of what happens inside. Thank you!

See the nfs-iostat utility in the nfs-utils package:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-iostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

Trond

#2 lore raw
Quoted 7 lines · Trond Myklebust
Quoted 2 lines · Andrey Borzenkov

I am grateful for any hints how can we monitor Linux NFS client and get real-life numbers of what happens inside. Thank you!

See the nfs-iostat utility in the nfs-utils package:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-iostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

Oh, and recent versions of 'sar' (from the 'sysstat' package) should also have support for NFS monitoring.

Trond

#3 lore raw
In reply to Trond Myklebust
Quoted 20 lines · On Sunday 12 of July 2009 02:44:27 Trond Myklebust wrote:
Quoted 19 lines · Andrey Borzenkov

Recently we have the case of very high latencies on NFS reads as reported by application (SAP R/3). NFS server was NetApp FAS; according to NetApp statistic, average volume read latencies were in order 10ms, while SAP stats gave 30-50ms. Systems were interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.

On advice of my colleague we changed sunrpc.tcp_slot_table_entries from default 16 to 128 which seemed to make situation much better - without changing load pattern of filer in any visible way.

Now, I can understand, why we observed much higher latency on system and why changing (what effectively is) queue depth helped. But I am totally frustrated that there does not appear to be *any* possibility to detect this situation on Linux side and to get a real numbers of real NFS IO latencies or number of requests waiting to be executed (and I do not even dream about per-mount point stats).

Quoted 8 lines
Quoted 2 lines

I am grateful for any hints how can we monitor Linux NFS client and get real-life numbers of what happens inside. Thank you!

See the nfs-iostat utility in the nfs-utils package:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-i
ostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

This looks exactly like what is required, except ...

cn1:~ # ./nfs-iostat.py 2
Traceback (most recent call last):
File "./nfs-iostat.py", line 559, in ?
iostat_command(prog)
File "./nfs-iostat.py", line 464, in iostat_command
mountstats = parse_stats_file('/proc/self/mountstats')
File "./nfs-iostat.py", line 434, in parse_stats_file
f = file(filename)
IOError: [Errno 2] No such file or directory: '/proc/self/mountstats'

Which minimum kernel version is required for it?

Thank you!

#4 lore raw
Quoted 42 lines · Andrey Borzenkov
Quoted 20 lines · On Sunday 12 of July 2009 02:44:27 Trond Myklebust wrote:
Quoted 19 lines · Andrey Borzenkov

Recently we have the case of very high latencies on NFS reads as reported by application (SAP R/3). NFS server was NetApp FAS; according to NetApp statistic, average volume read latencies were in order 10ms, while SAP stats gave 30-50ms. Systems were interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.

On advice of my colleague we changed sunrpc.tcp_slot_table_entries from default 16 to 128 which seemed to make situation much better - without changing load pattern of filer in any visible way.

Now, I can understand, why we observed much higher latency on system and why changing (what effectively is) queue depth helped. But I am totally frustrated that there does not appear to be *any* possibility to detect this situation on Linux side and to get a real numbers of real NFS IO latencies or number of requests waiting to be executed (and I do not even dream about per-mount point stats).

Quoted 8 lines
Quoted 2 lines

I am grateful for any hints how can we monitor Linux NFS client and get real-life numbers of what happens inside. Thank you!

See the nfs-iostat utility in the nfs-utils package:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-i
ostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

This looks exactly like what is required, except ...

cn1:~ # ./nfs-iostat.py 2
Traceback (most recent call last):
File "./nfs-iostat.py", line 559, in ?
iostat_command(prog)
File "./nfs-iostat.py", line 464, in iostat_command
mountstats = parse_stats_file('/proc/self/mountstats')
File "./nfs-iostat.py", line 434, in parse_stats_file
f = file(filename)
IOError: [Errno 2] No such file or directory: '/proc/self/mountstats'

Quoted 3 lines

Which minimum kernel version is required for it?

Thank you!

You need kernel 2.6.17 or newer.

Cheers
Trond