A few enhancements to mountstats.py
Subsystems (2)
cover A few enhancements to mountstats.py
The following patches add a couple of enhancements to mountstats.py. I also fixed a few bugs I encountered along the way. Highlights include:
- added support for -f/--file to allow stats to be parsed from an aritrary input file instead of /proc/self/mountstats
- added support for -S/--since to show just the changes that have occurred between the current and a previous set of statisics (works with and without the -f option)
- added support for -R/--raw to generate 'raw' statistics (i.e. in the same format as /proc/self/mountstats). It's intended to be used with the -f and -S options.
- implemented the ms-nfsstat command to generate client-side nfsstat-like statisics (only works with a single mountpoint)
My motivation for these changes was so that I could take various copies of /proc/self/mountstats and massage them into data that I could feed into the 'report' option of Dros's nfsometer tool for scenarios where it's not feasible to run nfsometer itself (e.g. systems where we can't start with an 'idle' state (i.e. no NFS filesystems initially mounted), systems with multiple NFS filesystems mounted, and workloads that can't easily be boiled down into an nfsometer workload file or run via the custom workload environment variables).
Scott Mayhew (15):
mountstats: Fix up NFS event counters
mountstats: Add lists of various counters
mountstats: Refactor __parse_nfs_line and __parse_rpc_line
mountstats: Refactor compare_iostats
mountstats: Convert existing option parsing to use the getopt module
mountstats: Make ms-iostat output match that of nfs-iostat.py
mountstats: Make print_iostat_summary handle newly appearing mounts
mountstats: Add support for -f/--file to the mountstats and ms-iostat
commands
mountstats: Add support for -S/--since to the mountstats and ms-iostat
commands
mountstats: Fix IndexError in __parse_nfs_line
mountstats: Allow mountstats_command to take a variable number of
mountpoints
mountstats: Add support for -R/--raw to mountstats_command
mountstats: Implement nfsstat_command
mountstats: Remove the --start and --end options
mountstats: Update the help output
tools/mountstats/mountstats.py | 730 ++++++++++++++++++++++++++++++----------- 1 file changed, 544 insertions(+), 186 deletions(-) mode change 100644 => 100755 tools/mountstats/mountstats.py
Signature · 1 line
1.9.3
RFC PATCH 1/15 mountstats: Fix up NFS event counters
The event counters in the mountstats program aren't in sync with the event counters in the kernel. Removed syncinodes and added vfsupdatepage, vfssetattr, congestionwait, pnfsreads, and pnfswrites.
--- tools/mountstats/mountstats.py | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) mode change 100644 => 100755 tools/mountstats/mountstats.py
tools/mountstats/mountstats.py +18 -15
@@ -73,25 +73,29 @@ class DeviceData: self.__nfs_data['dentryrevalidates'] = int(words[2]) self.__nfs_data['datainvalidates'] = int(words[3]) self.__nfs_data['attrinvalidates'] = int(words[4]) - self.__nfs_data['syncinodes'] = int(words[5]) - self.__nfs_data['vfsopen'] = int(words[6]) - self.__nfs_data['vfslookup'] = int(words[7]) - self.__nfs_data['vfspermission'] = int(words[8]) + self.__nfs_data['vfsopen'] = int(words[5]) + self.__nfs_data['vfslookup'] = int(words[6]) + self.__nfs_data['vfspermission'] = int(words[7]) + self.__nfs_data['vfsupdatepage'] = int(words[8]) self.__nfs_data['vfsreadpage'] = int(words[9]) self.__nfs_data['vfsreadpages'] = int(words[10]) self.__nfs_data['vfswritepage'] = int(words[11]) self.__nfs_data['vfswritepages'] = int(words[12]) self.__nfs_data['vfsreaddir'] = int(words[13]) - self.__nfs_data['vfsflush'] = int(words[14]) - self.__nfs_data['vfsfsync'] = int(words[15]) - self.__nfs_data['vfslock'] = int(words[16]) - self.__nfs_data['vfsrelease'] = int(words[17]) - self.__nfs_data['setattrtrunc'] = int(words[18]) - self.__nfs_data['extendwrite'] = int(words[19]) - self.__nfs_data['sillyrenames'] = int(words[20]) - self.__nfs_data['shortreads'] = int(words[21]) - self.__nfs_data['shortwrites'] = int(words[22]) - self.__nfs_data['delay'] = int(words[23]) + self.__nfs_data['vfssetattr'] = int(words[14]) + self.__nfs_data['vfsflush'] = int(words[15]) + self.__nfs_data['vfsfsync'] = int(words[16]) + self.__nfs_data['vfslock'] = int(words[17]) + self.__nfs_data['vfsrelease'] = int(words[18]) + self.__nfs_data['congestionwait'] = int(words[19]) + self.__nfs_data['setattrtrunc'] = int(words[20]) + self.__nfs_data['extendwrite'] = int(words[21]) + self.__nfs_data['sillyrenames'] = int(words[22]) + self.__nfs_data['shortreads'] = int(words[23]) + self.__nfs_data['shortwrites'] = int(words[24]) + self.__nfs_data['delay'] = int(words[25]) + self.__nfs_data['pnfsreads'] = int(words[26]) + self.__nfs_data['pnfswrites'] = int(words[27]) elif words[0] == 'bytes:': self.__nfs_data['normalreadbytes'] = int(words[1]) self.__nfs_data['normalwritebytes'] = int(words[2])
@@ -201,7 +205,6 @@ class DeviceData: print('Cache events:') print(' data cache invalidated %d times' % self.__nfs_data['datainvalidates']) print(' attribute cache invalidated %d times' % self.__nfs_data['attrinvalidates']) - print(' inodes synced %d times' % self.__nfs_data['syncinodes']) print() print('VFS calls:') print(' VFS requested %d inode revalidations' % self.__nfs_data['inoderevalidates'])
Signature · 1 line
1.9.3
RFC PATCH 2/15 mountstats: Add lists of various counters
In reply to Scott MayhewThe NfsEventCounters and NfsByteCounters were lifted straight from nfs-iostat.py. Also added counters for the xprt line (for udp, tcp, and rdma) as well as for v3 and v4 NFS ops. These will allow for more compact code in a couple of places.
--- tools/mountstats/mountstats.py | 168 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+)
tools/mountstats/mountstats.py +168 -0
@@ -32,6 +32,174 @@ def difference(x, y): """ return x - y +NfsEventCounters = [ + 'inoderevalidates', + 'dentryrevalidates', + 'datainvalidates', + 'attrinvalidates', + 'vfsopen', + 'vfslookup', + 'vfspermission', + 'vfsupdatepage', + 'vfsreadpage', + 'vfsreadpages', + 'vfswritepage', + 'vfswritepages', + 'vfsreaddir', + 'vfssetattr', + 'vfsflush', + 'vfsfsync', + 'vfslock', + 'vfsrelease', + 'congestionwait', + 'setattrtrunc', + 'extendwrite', + 'sillyrenames', + 'shortreads', + 'shortwrites', + 'delay', + 'pnfsreads', + 'pnfswrites' +] + +NfsByteCounters = [ + 'normalreadbytes', + 'normalwritebytes', + 'directreadbytes', + 'directwritebytes', + 'serverreadbytes', + 'serverwritebytes', + 'readpages', + 'writepages' +] + +XprtUdpCounters = [ + 'port', + 'bind_count', + 'rpcsends', + 'rpcreceives', + 'badxids', + 'inflightsends', + 'backlogutil' +] + +XprtTcpCounters = [ + 'port', + 'bind_count', + 'connect_count', + 'connect_time', + 'idle_time', + 'rpcsends', + 'rpcreceives', + 'badxids', + 'inflightsends', + 'backlogutil' +] + +XprtRdmaCounters = [ + 'port', + 'bind_count', + 'connect_count', + 'connect_time', + 'idle_time', + 'rpcsends', + 'rpcreceives', + 'badxids', + 'backlogutil', + 'read_chunks', + 'write_chunks', + 'reply_chunks', + 'total_rdma_req', + 'total_rdma_rep', + 'pullup', + 'fixup', + 'hardway', + 'failed_marshal', + 'bad_reply' +] + +Nfsv3ops = [ + 'NULL', + 'GETATTR', + 'SETATTR', + 'LOOKUP', + 'ACCESS', + 'READLINK', + 'READ', + 'WRITE', + 'CREATE', + 'MKDIR', + 'SYMLINK', + 'MKNOD', + 'REMOVE', + 'RMDIR', + 'RENAME', + 'LINK', + 'READDIR', + 'READDIRPLUS', + 'FSSTAT', + 'FSINFO', + 'PATHCONF', + 'COMMIT' +] + +Nfsv4ops = [ + 'NULL', + 'READ', + 'WRITE', + 'COMMIT', + 'OPEN', + 'OPEN_CONFIRM', + 'OPEN_NOATTR', + 'OPEN_DOWNGRADE', + 'CLOSE', + 'SETATTR', + 'FSINFO', + 'RENEW', + 'SETCLIENTID', + 'SETCLIENTID_CONFIRM', + 'LOCK', + 'LOCKT', + 'LOCKU', + 'ACCESS', + 'GETATTR', + 'LOOKUP', + 'LOOKUP_ROOT', + 'REMOVE', + 'RENAME', + 'LINK', + 'SYMLINK', + 'CREATE', + 'PATHCONF', + 'STATFS', + 'READLINK', + 'READDIR', + 'SERVER_CAPS', + 'DELEGRETURN', + 'GETACL', + 'SETACL', + 'FS_LOCATIONS', + 'RELEASE_LOCKOWNER', + 'SECINFO', + 'FSID_PRESENT', + 'EXCHANGE_ID', + 'CREATE_SESSION', + 'DESTROY_SESSION', + 'SEQUENCE', + 'GET_LEASE_TIME', + 'RECLAIM_COMPLETE', + 'LAYOUTGET', + 'GETDEVICEINFO', + 'LAYOUTCOMMIT', + 'LAYOUTRETURN', + 'SECINFO_NO_NAME', + 'TEST_STATEID', + 'FREE_STATEID', + 'GETDEVICELIST', + 'BIND_CONN_TO_SESSION', + 'DESTROY_CLIENTID' +] + class DeviceData: """DeviceData objects provide methods for parsing and displaying data for a single mount grabbed from /proc/self/mountstats
Signature · 1 line
1.9.3
RFC PATCH 3/15 mountstats: Refactor __parse_nfs_line and __parse_rpc_line
In reply to Scott MayhewIterate over the newly added counter lists instead of having a ton of assignment statements.
--- tools/mountstats/mountstats.py | 92 +++++++++++------------------------------- 1 file changed, 23 insertions(+), 69 deletions(-)
tools/mountstats/mountstats.py +23 -69
@@ -237,40 +237,18 @@ class DeviceData: if self.__nfs_data['flavor'] == 6: self.__nfs_data['pseudoflavor'] = int(keys[1].split('=')[1]) elif words[0] == 'events:': - self.__nfs_data['inoderevalidates'] = int(words[1]) - self.__nfs_data['dentryrevalidates'] = int(words[2]) - self.__nfs_data['datainvalidates'] = int(words[3]) - self.__nfs_data['attrinvalidates'] = int(words[4]) - self.__nfs_data['vfsopen'] = int(words[5]) - self.__nfs_data['vfslookup'] = int(words[6]) - self.__nfs_data['vfspermission'] = int(words[7]) - self.__nfs_data['vfsupdatepage'] = int(words[8]) - self.__nfs_data['vfsreadpage'] = int(words[9]) - self.__nfs_data['vfsreadpages'] = int(words[10]) - self.__nfs_data['vfswritepage'] = int(words[11]) - self.__nfs_data['vfswritepages'] = int(words[12]) - self.__nfs_data['vfsreaddir'] = int(words[13]) - self.__nfs_data['vfssetattr'] = int(words[14]) - self.__nfs_data['vfsflush'] = int(words[15]) - self.__nfs_data['vfsfsync'] = int(words[16]) - self.__nfs_data['vfslock'] = int(words[17]) - self.__nfs_data['vfsrelease'] = int(words[18]) - self.__nfs_data['congestionwait'] = int(words[19]) - self.__nfs_data['setattrtrunc'] = int(words[20]) - self.__nfs_data['extendwrite'] = int(words[21]) - self.__nfs_data['sillyrenames'] = int(words[22]) - self.__nfs_data['shortreads'] = int(words[23]) - self.__nfs_data['shortwrites'] = int(words[24]) - self.__nfs_data['delay'] = int(words[25]) - self.__nfs_data['pnfsreads'] = int(words[26]) - self.__nfs_data['pnfswrites'] = int(words[27]) + i = 1 + for key in NfsEventCounters: + try: + self.__nfs_data[key] = int(words[i]) + except IndexError as err: + self.__nfs_data[key] = 0 + i += 1 elif words[0] == 'bytes:': - self.__nfs_data['normalreadbytes'] = int(words[1]) - self.__nfs_data['normalwritebytes'] = int(words[2]) - self.__nfs_data['directreadbytes'] = int(words[3]) - self.__nfs_data['directwritebytes'] = int(words[4]) - self.__nfs_data['serverreadbytes'] = int(words[5]) - self.__nfs_data['serverwritebytes'] = int(words[6]) + i = 1 + for key in NfsByteCounters: + self.__nfs_data[key] = int(words[i]) + i += 1 def __parse_rpc_line(self, words): if words[0] == 'RPC':
@@ -279,44 +257,20 @@ class DeviceData: elif words[0] == 'xprt:': self.__rpc_data['protocol'] = words[1] if words[1] == 'udp': - self.__rpc_data['port'] = int(words[2]) - self.__rpc_data['bind_count'] = int(words[3]) - self.__rpc_data['rpcsends'] = int(words[4]) - self.__rpc_data['rpcreceives'] = int(words[5]) - self.__rpc_data['badxids'] = int(words[6]) - self.__rpc_data['inflightsends'] = int(words[7]) - self.__rpc_data['backlogutil'] = int(words[8]) + i = 2 + for key in XprtUdpCounters: + self.__rpc_data[key] = int(words[i]) + i += 1 elif words[1] == 'tcp': - self.__rpc_data['port'] = words[2] - self.__rpc_data['bind_count'] = int(words[3]) - self.__rpc_data['connect_count'] = int(words[4]) - self.__rpc_data['connect_time'] = int(words[5]) - self.__rpc_data['idle_time'] = int(words[6]) - self.__rpc_data['rpcsends'] = int(words[7]) - self.__rpc_data['rpcreceives'] = int(words[8]) - self.__rpc_data['badxids'] = int(words[9]) - self.__rpc_data['inflightsends'] = int(words[10]) - self.__rpc_data['backlogutil'] = int(words[11]) + i = 2 + for key in XprtTcpCounters: + self.__rpc_data[key] = int(words[i]) + i += 1 elif words[1] == 'rdma': - self.__rpc_data['port'] = words[2] - self.__rpc_data['bind_count'] = int(words[3]) - self.__rpc_data['connect_count'] = int(words[4]) - self.__rpc_data['connect_time'] = int(words[5]) - self.__rpc_data['idle_time'] = int(words[6]) - self.__rpc_data['rpcsends'] = int(words[7]) - self.__rpc_data['rpcreceives'] = int(words[8]) - self.__rpc_data['badxids'] = int(words[9]) - self.__rpc_data['backlogutil'] = int(words[10]) - self.__rpc_data['read_chunks'] = int(words[11]) - self.__rpc_data['write_chunks'] = int(words[12]) - self.__rpc_data['reply_chunks'] = int(words[13]) - self.__rpc_data['total_rdma_req'] = int(words[14]) - self.__rpc_data['total_rdma_rep'] = int(words[15]) - self.__rpc_data['pullup'] = int(words[16]) - self.__rpc_data['fixup'] = int(words[17]) - self.__rpc_data['hardway'] = int(words[18]) - self.__rpc_data['failed_marshal'] = int(words[19]) - self.__rpc_data['bad_reply'] = int(words[20]) + i = 2 + for key in XprtRdmaCounters: + self.__rpc_data[key] = int(words[i]) + i += 1 elif words[0] == 'per-op': self.__rpc_data['per-op'] = words else:
Signature · 1 line
1.9.3
RFC PATCH 4/15 mountstats: Refactor compare_iostats
In reply to Scott MayhewIterate over the newly added counters instead of using repeated assignment statements. Also compute the difference of every counter where it makes sense -- this will allow support for -S/--since to be implemented in the future.
--- tools/mountstats/mountstats.py | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
tools/mountstats/mountstats.py +19 -6
@@ -412,7 +412,11 @@ class DeviceData: def compare_iostats(self, old_stats): """Return the difference between two sets of stats """ + if old_stats.__nfs_data['age'] > self.__nfs_data['age']: + return self + result = DeviceData() + protocol = self.__rpc_data['protocol'] # copy self into result for key, value in self.__nfs_data.items():
@@ -427,12 +431,21 @@ class DeviceData: for op in result.__rpc_data['ops']: result.__rpc_data[op] = list(map(difference, self.__rpc_data[op], old_stats.__rpc_data[op])) - # update the remaining keys we care about - result.__rpc_data['rpcsends'] -= old_stats.__rpc_data['rpcsends'] - result.__rpc_data['backlogutil'] -= old_stats.__rpc_data['backlogutil'] - result.__nfs_data['serverreadbytes'] -= old_stats.__nfs_data['serverreadbytes'] - result.__nfs_data['serverwritebytes'] -= old_stats.__nfs_data['serverwritebytes'] - + # update the remaining keys + if protocol == 'udp': + for key in XprtUdpCounters: + result.__rpc_data[key] -= old_stats.__rpc_data[key] + elif protocol == 'tcp': + for key in XprtTcpCounters: + result.__rpc_data[key] -= old_stats.__rpc_data[key] + elif protocol == 'rdma': + for key in XprtRdmaCounters: + result.__rpc_data[key] -= old_stats.__rpc_data[key] + result.__nfs_data['age'] -= old_stats.__nfs_data['age'] + for key in NfsEventCounters: + result.__nfs_data[key] -= old_stats.__nfs_data[key] + for key in NfsByteCounters: + result.__nfs_data[key] -= old_stats.__nfs_data[key] return result def display_iostats(self, sample_time):
Signature · 1 line
1.9.3
RFC PATCH 5/15 mountstats: Convert existing option parsing to use the getopt module
In reply to Scott Mayhew--- tools/mountstats/mountstats.py | 62 +++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 31 deletions(-)
tools/mountstats/mountstats.py +31 -31
@@ -24,6 +24,7 @@ MA 02110-1301 USA """ import sys, os, time +import getopt Mountstats_version = '0.2'
@@ -547,37 +548,33 @@ def print_mountstats_help(name): def mountstats_command(): """Mountstats command """ + try: + opts, args = getopt.getopt(sys.argv[1:], "ehnrsv", ["end", "help", "nfs", "rpc", "start", "version"]) + except getopt.GetoptError as err: + print_mountstats_help(prog) + mountpoints = [] nfs_only = False rpc_only = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-e", "--end"): + raise Exception('Sampling is not yet implemented') + elif o in ("-h", "--help"): print_mountstats_help(prog) - return - - if arg in ['-v', '--version', 'version']: - print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) - - if arg in ['-n', '--nfs']: + elif o in ("-n", "--nfs"): nfs_only = True - continue - - if arg in ['-r', '--rpc']: + elif o in ("-r", "--rpc"): rpc_only = True - continue - - if arg in ['-s', '--start']: - raise Exception('Sampling is not yet implemented') - - if arg in ['-e', '--end']: + elif o in ("-s", "--start"): raise Exception('Sampling is not yet implemented') - - if arg == sys.argv[0]: - continue - - mountpoints += [arg] + elif o in ("-v", "--version"): + print('%s version %s' % (sys.argv[0], Mountstats_version)) + sys.exit(0) + else: + assert False, "unhandled option" + mountpoints += args if mountpoints == []: print_mountstats_help(prog)
@@ -662,23 +659,26 @@ def print_iostat_summary(old, new, devices, time): def iostat_command(): """iostat-like command for NFS mount points """ + try: + opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"]) + except getopt.GetoptError as err: + print_iostat_help(prog) mountstats = parse_stats_file('/proc/self/mountstats') devices = [] interval_seen = False count_seen = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-h", "--help"): print_iostat_help(prog) - return - - if arg in ['-v', '--version', 'version']: + sys.exit(0) + elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) - return - - if arg == sys.argv[0]: - continue + sys.exit(0) + else: + assert False, "unhandled option" + for arg in args: if arg in mountstats: devices += [arg] elif not interval_seen:
Signature · 1 line
1.9.3
Re: [nfs-utils RFC PATCH 05/15] mountstats: Convert existing option parsing to use the getopt module
On Nov 5, 2014, at 12:01 PM, Scott Mayhew <[email protected]> wrote:
I’m not clear why you need to replace this code. But if you really do need to replace it, why not use the more Python-esque argparse module, which would replace both the option parsing and help text?
Quoted 116 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 62 +++++++++++++++++++++---------------------1 file changed, 31 insertions(+), 31 deletions(-)tools/mountstats/mountstats.py +0
-0@@ -24,6 +24,7 @@ MA 02110-1301 USA"""
import sys, os, time
+import getopt
Mountstats_version= '0.2'tools/mountstats/mountstats.py +0
-0@@ -547,37 +548,33 @@ def print_mountstats_help(name):def
mountstats_command():"""Mountstats command """tools/mountstats/mountstats.py +18
-22+ try: + opts, args = getopt.getopt(sys.argv[1:], "ehnrsv", ["end", "help", "nfs", "rpc", "start", "version"]) + except getopt.GetoptError as err: + print_mountstats_help(prog) + mountpoints = [] nfs_only = False rpc_only = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-e", "--end"): + raise Exception('Sampling is not yet implemented') + elif o in ("-h", "--help"): print_mountstats_help(prog) - return - - if arg in ['-v', '--version', 'version']: - print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) - - if arg in ['-n', '--nfs']: + elif o in ("-n", "--nfs"): nfs_only = True - continue - - if arg in ['-r', '--rpc']: + elif o in ("-r", "--rpc"): rpc_only = True - continue - - if arg in ['-s', '--start']: - raise Exception('Sampling is not yet implemented') - - if arg in ['-e', '--end']: + elif o in ("-s", "--start"): raise Exception('Sampling is not yet implemented') - - if arg == sys.argv[0]: - continue - - mountpoints += [arg] + elif o in ("-v", "--version"): + print('%s version %s' % (sys.argv[0], Mountstats_version)) + sys.exit(0) + else: + assert False, "unhandled option" + mountpoints += args if mountpoints == []: print_mountstats_help(prog)@@ -662,23 +659,26 @@ def print_iostat_summary(old, new, devices, time):def
iostat_command():
"""iostat-like command for NFS mount points
"""tools/mountstats/mountstats.py +12
-9+ try: + opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"]) + except getopt.GetoptError as err: + print_iostat_help(prog) mountstats = parse_stats_file('/proc/self/mountstats') devices = [] interval_seen = False count_seen = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-h", "--help"): print_iostat_help(prog) - return - - if arg in ['-v', '--version', 'version']: + sys.exit(0) + elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) - return - - if arg == sys.argv[0]: - continue + sys.exit(0) + else: + assert False, "unhandled option" + for arg in args: if arg in mountstats: devices += [arg] elif not interval_seen:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 05/15] mountstats: Convert existing option parsing to use the getopt module
Quoted 7 lines · Chuck Lever
On Nov 5, 2014, at 12:01 PM, Scott Mayhew <[email protected]> wrote:
I?m not clear why you need to replace this code. But if you really do need to replace it, why not use the more Python-esque argparse module, which would replace both the option parsing and help text?
This started out as a hacked up version of the mountstats program that I was really planning on sending anywhere, and originally I had only done the nfsstat options using getopt. When I redid the changes in git I decided to convert the rest of them to use getopt as well. I can change it back to the original parsing or I can use argparse. Let me know which you prefer (I have no preference either way).
Quoted 124 lines
Quoted 117 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 62 +++++++++++++++++++++---------------------1 file changed, 31 insertions(+), 31 deletions(-)tools/mountstats/mountstats.py +0
-0@@ -24,6 +24,7 @@ MA 02110-1301 USA"""
import sys, os, time
+import getopt
Mountstats_version= '0.2'tools/mountstats/mountstats.py +0
-0@@ -547,37 +548,33 @@ def print_mountstats_help(name):def
mountstats_command():"""Mountstats command """tools/mountstats/mountstats.py +18
-22+ try: + opts, args = getopt.getopt(sys.argv[1:], "ehnrsv", ["end", "help", "nfs", "rpc", "start", "version"]) + except getopt.GetoptError as err: + print_mountstats_help(prog) + mountpoints = [] nfs_only = False rpc_only = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-e", "--end"): + raise Exception('Sampling is not yet implemented') + elif o in ("-h", "--help"): print_mountstats_help(prog) - return - - if arg in ['-v', '--version', 'version']: - print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) - - if arg in ['-n', '--nfs']: + elif o in ("-n", "--nfs"): nfs_only = True - continue - - if arg in ['-r', '--rpc']: + elif o in ("-r", "--rpc"): rpc_only = True - continue - - if arg in ['-s', '--start']: - raise Exception('Sampling is not yet implemented') - - if arg in ['-e', '--end']: + elif o in ("-s", "--start"): raise Exception('Sampling is not yet implemented') - - if arg == sys.argv[0]: - continue - - mountpoints += [arg] + elif o in ("-v", "--version"): + print('%s version %s' % (sys.argv[0], Mountstats_version)) + sys.exit(0) + else: + assert False, "unhandled option" + mountpoints += args if mountpoints == []: print_mountstats_help(prog)@@ -662,23 +659,26 @@ def print_iostat_summary(old, new, devices, time):def
iostat_command():
"""iostat-like command for NFS mount points
"""tools/mountstats/mountstats.py +12
-9+ try: + opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"]) + except getopt.GetoptError as err: + print_iostat_help(prog) mountstats = parse_stats_file('/proc/self/mountstats') devices = [] interval_seen = False count_seen = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-h", "--help"): print_iostat_help(prog) - return - - if arg in ['-v', '--version', 'version']: + sys.exit(0) + elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) - return - - if arg == sys.argv[0]: - continue + sys.exit(0) + else: + assert False, "unhandled option" + for arg in args: if arg in mountstats: devices += [arg] elif not interval_seen:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 05/15] mountstats: Convert existing option parsing to use the getopt module
Quoted 10 lines · Scott Mayhew
Quoted 7 lines · Chuck Lever
On Nov 5, 2014, at 12:01 PM, Scott Mayhew <[email protected]> wrote:
I?m not clear why you need to replace this code. But if you really do need to replace it, why not use the more Python-esque argparse module, which would replace both the option parsing and help text?
Quoted 6 lines
This started out as a hacked up version of the mountstats program that I was really planning on sending anywhere, and originally I had only done the nfsstat options using getopt. When I redid the changes in git I decided to convert the rest of them to use getopt as well. I can change it back to the original parsing or I can use argparse. Let me know which you prefer (I have no preference either way).
I wrote mountstats almost 10 years ago, and Python has changed a lot since then. It makes sense to modernize this a bit. I think argparse:
+ is more idiomatic Python
+ allows you to fix up the help text and attach it directly to each option
+ maybe makes it easier to manage the coexistence of the different commands in the same source code?
Give it a try. Here’s some sample code:
If it starts looking crazy, then stick with getopt.
Quoted 113 lines
Quoted 113 lines
Quoted 113 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 62 +++++++++++++++++++++---------------------1 file changed, 31 insertions(+), 31 deletions(-)tools/mountstats/mountstats.py +0
-0@@ -24,6 +24,7 @@ MA 02110-1301 USA"""
import sys, os, time
+import getopt
Mountstats_version= '0.2'tools/mountstats/mountstats.py +0
-0@@ -547,37 +548,33 @@ def print_mountstats_help(name):def
mountstats_command():"""Mountstats command """tools/mountstats/mountstats.py +18
-22+ try: + opts, args = getopt.getopt(sys.argv[1:], "ehnrsv", ["end", "help", "nfs", "rpc", "start", "version"]) + except getopt.GetoptError as err: + print_mountstats_help(prog) + mountpoints = [] nfs_only = False rpc_only = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-e", "--end"): + raise Exception('Sampling is not yet implemented') + elif o in ("-h", "--help"): print_mountstats_help(prog) - return - - if arg in ['-v', '--version', 'version']: - print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) - - if arg in ['-n', '--nfs']: + elif o in ("-n", "--nfs"): nfs_only = True - continue - - if arg in ['-r', '--rpc']: + elif o in ("-r", "--rpc"): rpc_only = True - continue - - if arg in ['-s', '--start']: - raise Exception('Sampling is not yet implemented') - - if arg in ['-e', '--end']: + elif o in ("-s", "--start"): raise Exception('Sampling is not yet implemented') - - if arg == sys.argv[0]: - continue - - mountpoints += [arg] + elif o in ("-v", "--version"): + print('%s version %s' % (sys.argv[0], Mountstats_version)) + sys.exit(0) + else: + assert False, "unhandled option" + mountpoints += args if mountpoints == []: print_mountstats_help(prog)@@ -662,23 +659,26 @@ def print_iostat_summary(old, new, devices, time):def
iostat_command():
"""iostat-like command for NFS mount points
"""tools/mountstats/mountstats.py +12
-9+ try: + opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"]) + except getopt.GetoptError as err: + print_iostat_help(prog) mountstats = parse_stats_file('/proc/self/mountstats') devices = [] interval_seen = False count_seen = False - for arg in sys.argv: - if arg in ['-h', '--help', 'help', 'usage']: + for o, a in opts: + if o in ("-h", "--help"): print_iostat_help(prog) - return - - if arg in ['-v', '--version', 'version']: + sys.exit(0) + elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) - return - - if arg == sys.argv[0]: - continue + sys.exit(0) + else: + assert False, "unhandled option" + for arg in args: if arg in mountstats: devices += [arg] elif not interval_seen:Signature · 1 line
1.9.3
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
RFC PATCH 6/15 mountstats: Make ms-iostat output match that of nfs-iostat.py
In reply to Scott MayhewChanges mostly lifted straight from nfs-iostat.py.
--- tools/mountstats/mountstats.py | 104 ++++++++++++++++++++++++----------------- 1 file changed, 61 insertions(+), 43 deletions(-)
tools/mountstats/mountstats.py +61 -43
@@ -449,60 +449,78 @@ class DeviceData: result.__nfs_data[key] -= old_stats.__nfs_data[key] return result + def __print_rpc_op_stats(self, op, sample_time): + """Print generic stats for one RPC op + """ + if op not in self.__rpc_data: + return + + rpc_stats = self.__rpc_data[op] + ops = float(rpc_stats[0]) + retrans = float(rpc_stats[1] - rpc_stats[0]) + kilobytes = float(rpc_stats[3] + rpc_stats[4]) / 1024 + rtt = float(rpc_stats[6]) + exe = float(rpc_stats[7]) + + # prevent floating point exceptions + if ops != 0: + kb_per_op = kilobytes / ops + retrans_percent = (retrans * 100) / ops + rtt_per_op = rtt / ops + exe_per_op = exe / ops + else: + kb_per_op = 0.0 + retrans_percent = 0.0 + rtt_per_op = 0.0 + exe_per_op = 0.0 + + op += ':' + print(format(op.lower(), '<16s'), end='') + print(format('ops/s', '>8s'), end='') + print(format('kB/s', '>16s'), end='') + print(format('kB/op', '>16s'), end='') + print(format('retrans', '>16s'), end='') + print(format('avg RTT (ms)', '>16s'), end='') + print(format('avg exe (ms)', '>16s')) + + print(format((ops / sample_time), '>24.3f'), end='') + print(format((kilobytes / sample_time), '>16.3f'), end='') + print(format(kb_per_op, '>16.3f'), end='') + retransmits = '{0:>10.0f} ({1:>3.1f}%)'.format(retrans, retrans_percent).strip() + print(format(retransmits, '>16'), end='') + print(format(rtt_per_op, '>16.3f'), end='') + print(format(exe_per_op, '>16.3f')) + def display_iostats(self, sample_time): """Display NFS and RPC stats in an iostat-like way """ sends = float(self.__rpc_data['rpcsends']) if sample_time == 0: sample_time = float(self.__nfs_data['age']) + # sample_time could still be zero if the export was just mounted. + # Set it to 1 to avoid divide by zero errors in this case since we'll + # likely still have relevant mount statistics to show. + # + if sample_time == 0: + sample_time = 1; + if sends != 0: + backlog = (float(self.__rpc_data['backlogutil']) / sends) / sample_time + else: + backlog = 0.0 print() print('%s mounted on %s:' % \ (self.__nfs_data['export'], self.__nfs_data['mountpoint'])) + print() - print('\top/s\trpc bklog') - print('\t%.2f' % (sends / sample_time), end=' ') - if sends != 0: - print('\t%.2f' % \ - ((float(self.__rpc_data['backlogutil']) / sends) / sample_time)) - else: - print('\t0.00') - - # reads: ops/s, kB/s, avg rtt, and avg exe - # XXX: include avg xfer size and retransmits? - read_rpc_stats = self.__rpc_data['READ'] - ops = float(read_rpc_stats[0]) - kilobytes = float(self.__nfs_data['serverreadbytes']) / 1024 - rtt = float(read_rpc_stats[6]) - exe = float(read_rpc_stats[7]) - - print('\treads:\tops/s\t\tkB/s\t\tavg RTT (ms)\tavg exe (ms)') - print('\t\t%.2f' % (ops / sample_time), end=' ') - print('\t\t%.2f' % (kilobytes / sample_time), end=' ') - if ops != 0: - print('\t\t%.2f' % (rtt / ops), end=' ') - print('\t\t%.2f' % (exe / ops)) - else: - print('\t\t0.00', end=' ') - print('\t\t0.00') - - # writes: ops/s, kB/s, avg rtt, and avg exe - # XXX: include avg xfer size and retransmits? - write_rpc_stats = self.__rpc_data['WRITE'] - ops = float(write_rpc_stats[0]) - kilobytes = float(self.__nfs_data['serverwritebytes']) / 1024 - rtt = float(write_rpc_stats[6]) - exe = float(write_rpc_stats[7]) - - print('\twrites:\tops/s\t\tkB/s\t\tavg RTT (ms)\tavg exe (ms)') - print('\t\t%.2f' % (ops / sample_time), end=' ') - print('\t\t%.2f' % (kilobytes / sample_time), end=' ') - if ops != 0: - print('\t\t%.2f' % (rtt / ops), end=' ') - print('\t\t%.2f' % (exe / ops)) - else: - print('\t\t0.00', end=' ') - print('\t\t0.00') + print(format('ops/s', '>16') + format('rpc bklog', '>16')) + print(format((sends / sample_time), '>16.3f'), end='') + print(format(backlog, '>16.3f')) + print() + + self.__print_rpc_op_stats('READ', sample_time) + self.__print_rpc_op_stats('WRITE', sample_time) + sys.stdout.flush() def parse_stats_file(filename): """pop the contents of a mountstats file into a dictionary,
Signature · 1 line
1.9.3
RFC PATCH 7/15 mountstats: Make print_iostat_summary handle newly appearing mounts
In reply to Scott Mayhew--- tools/mountstats/mountstats.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/mountstats/mountstats.py +1 -1
@@ -666,7 +666,7 @@ def print_iostat_summary(old, new, devices, time): for device in devices: stats = DeviceData() stats.parse_stats(new[device]) - if not old: + if not old or device not in old: stats.display_iostats(time) else: old_stats = DeviceData()
Signature · 1 line
1.9.3
RFC PATCH 8/15 mountstats: Add support for -f/--file to the mountstats and ms-iostat commands
In reply to Scott MayhewAdd support for the -f/--file option to allow parsing of data from an arbitrary file instead of /proc/self/mountstats.
--- tools/mountstats/mountstats.py | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-)
tools/mountstats/mountstats.py +28 -7
@@ -557,6 +557,7 @@ def print_mountstats_help(name): print(' Display NFS client per-mount statistics.') print() print(' --version display the version of this command') + print(' --file <file> read stats from \'file\' instead of /proc/self/mountstats') print(' --nfs display only the NFS statistics') print(' --rpc display only the RPC statistics') print(' --start sample and save statistics')
@@ -567,17 +568,20 @@ def mountstats_command(): """Mountstats command """ try: - opts, args = getopt.getopt(sys.argv[1:], "ehnrsv", ["end", "help", "nfs", "rpc", "start", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsv", ["end", "file=", "help", "nfs", "rpc", "start", "version"]) except getopt.GetoptError as err: print_mountstats_help(prog) mountpoints = [] nfs_only = False rpc_only = False + infile = None for o, a in opts: if o in ("-e", "--end"): raise Exception('Sampling is not yet implemented') + elif o in ("-f", "--file"): + infile = a elif o in ("-h", "--help"): print_mountstats_help(prog) sys.exit(0)
@@ -602,7 +606,9 @@ def mountstats_command(): print_mountstats_help(prog) return - mountstats = parse_stats_file('/proc/self/mountstats') + if not infile: + infile = '/proc/self/mountstats' + mountstats = parse_stats_file(infile) for mp in mountpoints: if mp not in mountstats:
@@ -678,16 +684,20 @@ def iostat_command(): """iostat-like command for NFS mount points """ try: - opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hv", ["file=", "help", "version"]) except getopt.GetoptError as err: print_iostat_help(prog) - mountstats = parse_stats_file('/proc/self/mountstats') devices = [] interval_seen = False count_seen = False + infile_seen = False + infile = None for o, a in opts: - if o in ("-h", "--help"): + if o in ("-f", "--file"): + infile = a + infile_seen = True + elif o in ("-h", "--help"): print_iostat_help(prog) sys.exit(0) elif o in ("-v", "--version"):
@@ -695,6 +705,9 @@ def iostat_command(): sys.exit(0) else: assert False, "unhandled option" + if not infile: + infile = '/proc/self/mountstats' + mountstats = parse_stats_file(infile) for arg in args: if arg in mountstats:
@@ -702,14 +715,22 @@ def iostat_command(): elif not interval_seen: interval = int(arg) if interval > 0: - interval_seen = True + if infile_seen: + print('interval may not be used with the -f option') + return + else: + interval_seen = True else: print('Illegal <interval> value') return elif not count_seen: count = int(arg) if count > 0: - count_seen = True + if infile_seen: + print('count may not be used with the -f option') + return + else: + count_seen = True else: print('Illegal <count> value') return
Signature · 1 line
1.9.3
RFC PATCH 9/15 mountstats: Add support for -S/--since to the mountstats and ms-iostat commands
In reply to Scott MayhewAdd support for -S/--since to the mountstats and ms-iostat commands to display the delta between a previous copy of /proc/self/mountstats and the current /proc/self/mountstats file. Can be combined with the -f option to show the statistics between two different points in time.
--- tools/mountstats/mountstats.py | 61 +++++++++++++++++++++++++++++------------- 1 file changed, 43 insertions(+), 18 deletions(-)
tools/mountstats/mountstats.py +43 -18
@@ -562,13 +562,28 @@ def print_mountstats_help(name): print(' --rpc display only the RPC statistics') print(' --start sample and save statistics') print(' --end resample statistics and compare them with saved') + print(' --since <file> shows difference between current stats and those in \'file\'') print() +def print_mountstats(stats, nfs_only, rpc_only): + if nfs_only: + stats.display_nfs_options() + stats.display_nfs_events() + stats.display_nfs_bytes() + elif rpc_only: + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() + else: + stats.display_nfs_options() + stats.display_nfs_bytes() + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() + def mountstats_command(): """Mountstats command """ try: - opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsv", ["end", "file=", "help", "nfs", "rpc", "start", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since="]) except getopt.GetoptError as err: print_mountstats_help(prog)
@@ -576,6 +591,7 @@ def mountstats_command(): nfs_only = False rpc_only = False infile = None + since = None for o, a in opts: if o in ("-e", "--end"):
@@ -594,6 +610,8 @@ def mountstats_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" mountpoints += args
@@ -610,6 +628,9 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + if since: + old_mountstats = parse_stats_file(since) + for mp in mountpoints: if mp not in mountstats: print('Statistics for mount point %s not found' % mp)
@@ -622,18 +643,15 @@ def mountstats_command(): print('Mount point %s exists but is not an NFS mount' % mp) continue - if nfs_only: - stats.display_nfs_options() - stats.display_nfs_events() - stats.display_nfs_bytes() - elif rpc_only: - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + if not since: + print_mountstats(stats, nfs_only, rpc_only) + elif since and mp not in old_mountstats: + print_mountstats(stats, nfs_only, rpc_only) else: - stats.display_nfs_options() - stats.display_nfs_bytes() - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + old_stats = DeviceData() + old_stats.parse_stats(old_mountstats[mp]) + diff_stats = stats.compare_iostats(old_stats) + print_mountstats(diff_stats, nfs_only, rpc_only) def print_nfsstat_help(name): print('usage: %s [ options ]' % name)
@@ -684,7 +702,7 @@ def iostat_command(): """iostat-like command for NFS mount points """ try: - opts, args = getopt.getopt(sys.argv[1:], "f:hv", ["file=", "help", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hvS:", ["file=", "help", "version", "since="]) except getopt.GetoptError as err: print_iostat_help(prog) devices = []
@@ -692,6 +710,7 @@ def iostat_command(): count_seen = False infile_seen = False infile = None + since = None for o, a in opts: if o in ("-f", "--file"):
@@ -703,6 +722,8 @@ def iostat_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" if not infile:
@@ -715,8 +736,8 @@ def iostat_command(): elif not interval_seen: interval = int(arg) if interval > 0: - if infile_seen: - print('interval may not be used with the -f option') + if infile_seen or since: + print('interval may not be used with the -f or -S options') return else: interval_seen = True
@@ -726,8 +747,8 @@ def iostat_command(): elif not count_seen: count = int(arg) if count > 0: - if infile_seen: - print('count may not be used with the -f option') + if infile_seen or since: + print('count may not be used with the -f or -S options') return else: count_seen = True
@@ -735,6 +756,11 @@ def iostat_command(): print('Illegal <count> value') return + if since: + old_mountstats = parse_stats_file(since) + else: + old_mountstats = None + # make certain devices contains only NFS mount points if len(devices) > 0: check = []
@@ -754,7 +780,6 @@ def iostat_command(): print('No NFS mount points were found') return - old_mountstats = None sample_time = 0 if not interval_seen:
Signature · 1 line
1.9.3
Re: [nfs-utils RFC PATCH 09/15] mountstats: Add support for -S/--since to the mountstats and ms-iostat commands
Quoted 4 lines · Scott Mayhew
Add support for -S/--since to the mountstats and ms-iostat commands to display the delta between a previous copy of
/proc/self/mountstatsand the current/proc/self/mountstatsfile. Can be combined with the -f option to show the statistics between two different points in time.
This is exactly what —start and —send were supposed to do. Why not use use them?
Quoted 168 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 61 +++++++++++++++++++++++++++++-------------1 file changed, 43 insertions(+), 18 deletions(-)tools/mountstats/mountstats.py +15
-0@@ -562,13 +562,28 @@ def print_mountstats_help(name): print(' --rpc display only the RPC statistics') print(' --start sample and save statistics') print(' --end resample statistics and compare them with saved') + print(' --since <file> shows difference between current stats and those in \'file\'') print() +def print_mountstats(stats, nfs_only, rpc_only): + if nfs_only: + stats.display_nfs_options() + stats.display_nfs_events() + stats.display_nfs_bytes() + elif rpc_only: + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() + else: + stats.display_nfs_options() + stats.display_nfs_bytes() + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() +def
mountstats_command():"""Mountstats command """ try:tools/mountstats/mountstats.py +15
-12- opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsv", ["end", "file=", "help", "nfs", "rpc", "start", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since="]) except getopt.GetoptError as err: print_mountstats_help(prog)@@ -576,6 +591,7 @@ def mountstats_command(): nfs_only = False rpc_only = False infile = None + since = None for o, a in opts: if o in ("-e", "--end"):@@ -594,6 +610,8 @@ def mountstats_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" mountpoints += args@@ -610,6 +628,9 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + if since: + old_mountstats = parse_stats_file(since) + for mp in mountpoints: if mp not in mountstats: print('Statistics for mount point %s not found' % mp)@@ -622,18 +643,15 @@ def mountstats_command(): print('Mount point %s exists but is not an NFS mount' % mp) continue - if nfs_only: - stats.display_nfs_options() - stats.display_nfs_events() - stats.display_nfs_bytes() - elif rpc_only: - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + if not since: + print_mountstats(stats, nfs_only, rpc_only) + elif since and mp not in old_mountstats: + print_mountstats(stats, nfs_only, rpc_only) else: - stats.display_nfs_options() - stats.display_nfs_bytes() - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + old_stats = DeviceData() + old_stats.parse_stats(old_mountstats[mp]) + diff_stats = stats.compare_iostats(old_stats) + print_mountstats(diff_stats, nfs_only, rpc_only)def
print_nfsstat_help(name):
print('usage: %s [ options ]' % name)tools/mountstats/mountstats.py +13
-6@@ -684,7 +702,7 @@ def iostat_command(): """iostat-like command for NFS mount points """ try: - opts, args = getopt.getopt(sys.argv[1:], "f:hv", ["file=", "help", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hvS:", ["file=", "help", "version", "since="]) except getopt.GetoptError as err: print_iostat_help(prog) devices = []@@ -692,6 +710,7 @@ def iostat_command(): count_seen = False infile_seen = False infile = None + since = None for o, a in opts: if o in ("-f", "--file"):@@ -703,6 +722,8 @@ def iostat_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" if not infile:@@ -715,8 +736,8 @@ def iostat_command(): elif not interval_seen: interval = int(arg) if interval > 0: - if infile_seen: - print('interval may not be used with the -f option') + if infile_seen or since: + print('interval may not be used with the -f or -S options') return else: interval_seen = True@@ -726,8 +747,8 @@ def iostat_command(): elif not count_seen: count = int(arg) if count > 0: - if infile_seen: - print('count may not be used with the -f option') + if infile_seen or since: + print('count may not be used with the -f or -S options') return else: count_seen = True@@ -735,6 +756,11 @@ def iostat_command(): print('Illegal <count> value') return + if since: + old_mountstats = parse_stats_file(since) + else: + old_mountstats = None + # make certain devices contains only NFS mount points if len(devices) > 0: check = []@@ -754,7 +780,6 @@ def iostat_command(): print('No NFS mount points were found') return - old_mountstats = None sample_time = 0 if not interval_seen:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 09/15] mountstats: Add support for -S/--since to the mountstats and ms-iostat commands
Quoted 10 lines · Chuck Lever
Quoted 4 lines · Scott Mayhew
Add support for -S/--since to the mountstats and ms-iostat commands to display the delta between a previous copy of
/proc/self/mountstatsand the current/proc/self/mountstatsfile. Can be combined with the -f option to show the statistics between two different points in time.This is exactly what ?start and ?send were supposed to do. Why not use use them?
The help output for --start and --end didn't show arguments, so I was under the impression that these were something you'd use on the same system where the stats are being generated (as opposed to capturing the raw data so you can generate the output on some other system).
Plus I was thinking that a neat use for --start and --end would be to use those with an interval to capture periodic stats that could be then used to generate line graphs. But I guess that could just as easily be done with cp or cat in cron job or a shell loop.
Finally I was mimicking the -S/--since option from nfsstat.c.
Quoted 176 lines
Quoted 169 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 61 +++++++++++++++++++++++++++++-------------1 file changed, 43 insertions(+), 18 deletions(-)tools/mountstats/mountstats.py +15
-0@@ -562,13 +562,28 @@ def print_mountstats_help(name): print(' --rpc display only the RPC statistics') print(' --start sample and save statistics') print(' --end resample statistics and compare them with saved') + print(' --since <file> shows difference between current stats and those in \'file\'') print() +def print_mountstats(stats, nfs_only, rpc_only): + if nfs_only: + stats.display_nfs_options() + stats.display_nfs_events() + stats.display_nfs_bytes() + elif rpc_only: + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() + else: + stats.display_nfs_options() + stats.display_nfs_bytes() + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() +def
mountstats_command():"""Mountstats command """ try:tools/mountstats/mountstats.py +15
-12- opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsv", ["end", "file=", "help", "nfs", "rpc", "start", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since="]) except getopt.GetoptError as err: print_mountstats_help(prog)@@ -576,6 +591,7 @@ def mountstats_command(): nfs_only = False rpc_only = False infile = None + since = None for o, a in opts: if o in ("-e", "--end"):@@ -594,6 +610,8 @@ def mountstats_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" mountpoints += args@@ -610,6 +628,9 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + if since: + old_mountstats = parse_stats_file(since) + for mp in mountpoints: if mp not in mountstats: print('Statistics for mount point %s not found' % mp)@@ -622,18 +643,15 @@ def mountstats_command(): print('Mount point %s exists but is not an NFS mount' % mp) continue - if nfs_only: - stats.display_nfs_options() - stats.display_nfs_events() - stats.display_nfs_bytes() - elif rpc_only: - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + if not since: + print_mountstats(stats, nfs_only, rpc_only) + elif since and mp not in old_mountstats: + print_mountstats(stats, nfs_only, rpc_only) else: - stats.display_nfs_options() - stats.display_nfs_bytes() - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + old_stats = DeviceData() + old_stats.parse_stats(old_mountstats[mp]) + diff_stats = stats.compare_iostats(old_stats) + print_mountstats(diff_stats, nfs_only, rpc_only)def
print_nfsstat_help(name):
print('usage: %s [ options ]' % name)tools/mountstats/mountstats.py +13
-6@@ -684,7 +702,7 @@ def iostat_command(): """iostat-like command for NFS mount points """ try: - opts, args = getopt.getopt(sys.argv[1:], "f:hv", ["file=", "help", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hvS:", ["file=", "help", "version", "since="]) except getopt.GetoptError as err: print_iostat_help(prog) devices = []@@ -692,6 +710,7 @@ def iostat_command(): count_seen = False infile_seen = False infile = None + since = None for o, a in opts: if o in ("-f", "--file"):@@ -703,6 +722,8 @@ def iostat_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" if not infile:@@ -715,8 +736,8 @@ def iostat_command(): elif not interval_seen: interval = int(arg) if interval > 0: - if infile_seen: - print('interval may not be used with the -f option') + if infile_seen or since: + print('interval may not be used with the -f or -S options') return else: interval_seen = True@@ -726,8 +747,8 @@ def iostat_command(): elif not count_seen: count = int(arg) if count > 0: - if infile_seen: - print('count may not be used with the -f option') + if infile_seen or since: + print('count may not be used with the -f or -S options') return else: count_seen = True@@ -735,6 +756,11 @@ def iostat_command(): print('Illegal <count> value') return + if since: + old_mountstats = parse_stats_file(since) + else: + old_mountstats = None + # make certain devices contains only NFS mount points if len(devices) > 0: check = []@@ -754,7 +780,6 @@ def iostat_command(): print('No NFS mount points were found') return - old_mountstats = None sample_time = 0 if not interval_seen:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 09/15] mountstats: Add support for -S/--since to the mountstats and ms-iostat commands
Quoted 10 lines · Scott Mayhew
Quoted 8 lines · Chuck Lever
Quoted 4 lines · Scott Mayhew
Add support for -S/--since to the mountstats and ms-iostat commands to display the delta between a previous copy of
/proc/self/mountstatsand the current/proc/self/mountstatsfile. Can be combined with the -f option to show the statistics between two different points in time.
Quoted 7 lines
Quoted 2 lines
This is exactly what ?start and ?send were supposed to do. Why not use use them?
The help output for --start and --end didn't show arguments, so I was under the impression that these were something you'd use on the same system where the stats are being generated (as opposed to capturing the raw data so you can generate the output on some other system).
Why can’t you just copy /proc/self/mountstats to a permanent file?
Quoted 4 lines
Plus I was thinking that a neat use for --start and --end would be to use those with an interval to capture periodic stats that could be then used to generate line graphs. But I guess that could just as easily be done with cp or cat in cron job or a shell loop.
That was the original purpose.
Quoted 1 line
Finally I was mimicking the -S/--since option from
nfsstat.c.
Then, sounds like there’s precedent for -S.
Quoted 165 lines
Quoted 165 lines
Quoted 165 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 61 +++++++++++++++++++++++++++++-------------1 file changed, 43 insertions(+), 18 deletions(-)tools/mountstats/mountstats.py +15
-0@@ -562,13 +562,28 @@ def print_mountstats_help(name): print(' --rpc display only the RPC statistics') print(' --start sample and save statistics') print(' --end resample statistics and compare them with saved') + print(' --since <file> shows difference between current stats and those in \'file\'') print() +def print_mountstats(stats, nfs_only, rpc_only): + if nfs_only: + stats.display_nfs_options() + stats.display_nfs_events() + stats.display_nfs_bytes() + elif rpc_only: + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() + else: + stats.display_nfs_options() + stats.display_nfs_bytes() + stats.display_rpc_generic_stats() + stats.display_rpc_op_stats() +def
mountstats_command():"""Mountstats command """ try:tools/mountstats/mountstats.py +15
-12- opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsv", ["end", "file=", "help", "nfs", "rpc", "start", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since="]) except getopt.GetoptError as err: print_mountstats_help(prog)@@ -576,6 +591,7 @@ def mountstats_command(): nfs_only = False rpc_only = False infile = None + since = None for o, a in opts: if o in ("-e", "--end"):@@ -594,6 +610,8 @@ def mountstats_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" mountpoints += args@@ -610,6 +628,9 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + if since: + old_mountstats = parse_stats_file(since) + for mp in mountpoints: if mp not in mountstats: print('Statistics for mount point %s not found' % mp)@@ -622,18 +643,15 @@ def mountstats_command(): print('Mount point %s exists but is not an NFS mount' % mp) continue - if nfs_only: - stats.display_nfs_options() - stats.display_nfs_events() - stats.display_nfs_bytes() - elif rpc_only: - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + if not since: + print_mountstats(stats, nfs_only, rpc_only) + elif since and mp not in old_mountstats: + print_mountstats(stats, nfs_only, rpc_only) else: - stats.display_nfs_options() - stats.display_nfs_bytes() - stats.display_rpc_generic_stats() - stats.display_rpc_op_stats() + old_stats = DeviceData() + old_stats.parse_stats(old_mountstats[mp]) + diff_stats = stats.compare_iostats(old_stats) + print_mountstats(diff_stats, nfs_only, rpc_only)def
print_nfsstat_help(name):
print('usage: %s [ options ]' % name)tools/mountstats/mountstats.py +13
-6@@ -684,7 +702,7 @@ def iostat_command(): """iostat-like command for NFS mount points """ try: - opts, args = getopt.getopt(sys.argv[1:], "f:hv", ["file=", "help", "version"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hvS:", ["file=", "help", "version", "since="]) except getopt.GetoptError as err: print_iostat_help(prog) devices = []@@ -692,6 +710,7 @@ def iostat_command(): count_seen = False infile_seen = False infile = None + since = None for o, a in opts: if o in ("-f", "--file"):@@ -703,6 +722,8 @@ def iostat_command(): elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0) + elif o in ("-S", "--since"): + since = a else: assert False, "unhandled option" if not infile:@@ -715,8 +736,8 @@ def iostat_command(): elif not interval_seen: interval = int(arg) if interval > 0: - if infile_seen: - print('interval may not be used with the -f option') + if infile_seen or since: + print('interval may not be used with the -f or -S options') return else: interval_seen = True@@ -726,8 +747,8 @@ def iostat_command(): elif not count_seen: count = int(arg) if count > 0: - if infile_seen: - print('count may not be used with the -f option') + if infile_seen or since: + print('count may not be used with the -f or -S options') return else: count_seen = True@@ -735,6 +756,11 @@ def iostat_command(): print('Illegal <count> value') return + if since: + old_mountstats = parse_stats_file(since) + else: + old_mountstats = None + # make certain devices contains only NFS mount points if len(devices) > 0: check = []@@ -754,7 +780,6 @@ def iostat_command(): print('No NFS mount points were found') return - old_mountstats = None sample_time = 0 if not interval_seen:Signature · 1 line
1.9.3
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
RFC PATCH 10/15 mountstats: Fix IndexError in __parse_nfs_line
In reply to Scott MayhewIf __parse_nfs_line is is called on a line that has 'fstype nfsd', it'll raise an IndexError trying to parse a nonexistent statvers entry.
--- tools/mountstats/mountstats.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/mountstats/mountstats.py +2 -2
@@ -215,13 +215,13 @@ class DeviceData: self.__nfs_data['export'] = words[1] self.__nfs_data['mountpoint'] = words[4] self.__nfs_data['fstype'] = words[7] - if words[7].find('nfs') != -1: + if words[7].find('nfs') != -1 and words[7] != 'nfsd': self.__nfs_data['statvers'] = words[8] elif 'nfs' in words or 'nfs4' in words: self.__nfs_data['export'] = words[0] self.__nfs_data['mountpoint'] = words[3] self.__nfs_data['fstype'] = words[6] - if words[6].find('nfs') != -1: + if words[6].find('nfs') != -1 and words[6] != 'nfsd': self.__nfs_data['statvers'] = words[7] elif words[0] == 'age:': self.__nfs_data['age'] = int(words[1])
Signature · 1 line
1.9.3
RFC PATCH 11/15 mountstats: Allow mountstats_command to take a variable number of mountpoints
In reply to Scott MayhewAllow the mountstats command to take a variable number of mountpoints (including none, in which case it will print stats for all NFS mountpoints it finds).
--- tools/mountstats/mountstats.py | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
tools/mountstats/mountstats.py +19 -13
@@ -616,10 +616,6 @@ def mountstats_command(): assert False, "unhandled option" mountpoints += args - if mountpoints == []: - print_mountstats_help(prog) - return - if rpc_only == True and nfs_only == True: print_mountstats_help(prog) return
@@ -628,21 +624,31 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + # make certain devices contains only NFS mount points + if len(mountpoints) > 0: + check = [] + for device in mountpoints: + stats = DeviceData() + stats.parse_stats(mountstats[device]) + if stats.is_nfs_mountpoint(): + check += [device] + mountpoints = check + else: + for device, descr in mountstats.items(): + stats = DeviceData() + stats.parse_stats(descr) + if stats.is_nfs_mountpoint(): + mountpoints += [device] + if len(mountpoints) == 0: + print('No NFS mount points were found') + return + if since: old_mountstats = parse_stats_file(since) for mp in mountpoints: - if mp not in mountstats: - print('Statistics for mount point %s not found' % mp) - continue - stats = DeviceData() stats.parse_stats(mountstats[mp]) - - if not stats.is_nfs_mountpoint(): - print('Mount point %s exists but is not an NFS mount' % mp) - continue - if not since: print_mountstats(stats, nfs_only, rpc_only) elif since and mp not in old_mountstats:
Signature · 1 line
1.9.3
Re: [nfs-utils RFC PATCH 11/15] mountstats: Allow mountstats_command to take a variable number of mountpoints
Quoted 3 lines · Scott Mayhew
Allow the mountstats command to take a variable number of mountpoints (including none, in which case it will print stats for all NFS mountpoints it finds).
Cool idea.
Quoted 68 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 32 +++++++++++++++++++-------------1 file changed, 19 insertions(+), 13 deletions(-)tools/mountstats/mountstats.py +19
-13@@ -616,10 +616,6 @@ def mountstats_command(): assert False, "unhandled option" mountpoints += args - if mountpoints == []: - print_mountstats_help(prog) - return - if rpc_only == True and nfs_only == True: print_mountstats_help(prog) return@@ -628,21 +624,31 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + # make certain devices contains only NFS mount points + if len(mountpoints) > 0: + check = [] + for device in mountpoints: + stats = DeviceData() + stats.parse_stats(mountstats[device]) + if stats.is_nfs_mountpoint(): + check += [device] + mountpoints = check + else: + for device, descr in mountstats.items(): + stats = DeviceData() + stats.parse_stats(descr) + if stats.is_nfs_mountpoint(): + mountpoints += [device] + if len(mountpoints) == 0: + print('No NFS mount points were found') + return + if since: old_mountstats = parse_stats_file(since) for mp in mountpoints: - if mp not in mountstats: - print('Statistics for mount point %s not found' % mp) - continue - stats = DeviceData() stats.parse_stats(mountstats[mp]) - - if not stats.is_nfs_mountpoint(): - print('Mount point %s exists but is not an NFS mount' % mp) - continue - if not since: print_mountstats(stats, nfs_only, rpc_only) elif since and mp not in old_mountstats:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 11/15] mountstats: Allow mountstats_command to take a variable number of mountpoints
Quoted 8 lines · Chuck Lever
Quoted 3 lines · Scott Mayhew
Allow the mountstats command to take a variable number of mountpoints (including none, in which case it will print stats for all NFS mountpoints it finds).
Cool idea.
Thanks! It looks like I need to add some sort of a header when only the rpc stats are being printed though.
Quoted 76 lines
Quoted 69 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 32 +++++++++++++++++++-------------1 file changed, 19 insertions(+), 13 deletions(-)tools/mountstats/mountstats.py +19
-13@@ -616,10 +616,6 @@ def mountstats_command(): assert False, "unhandled option" mountpoints += args - if mountpoints == []: - print_mountstats_help(prog) - return - if rpc_only == True and nfs_only == True: print_mountstats_help(prog) return@@ -628,21 +624,31 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + # make certain devices contains only NFS mount points + if len(mountpoints) > 0: + check = [] + for device in mountpoints: + stats = DeviceData() + stats.parse_stats(mountstats[device]) + if stats.is_nfs_mountpoint(): + check += [device] + mountpoints = check + else: + for device, descr in mountstats.items(): + stats = DeviceData() + stats.parse_stats(descr) + if stats.is_nfs_mountpoint(): + mountpoints += [device] + if len(mountpoints) == 0: + print('No NFS mount points were found') + return + if since: old_mountstats = parse_stats_file(since) for mp in mountpoints: - if mp not in mountstats: - print('Statistics for mount point %s not found' % mp) - continue - stats = DeviceData() stats.parse_stats(mountstats[mp]) - - if not stats.is_nfs_mountpoint(): - print('Mount point %s exists but is not an NFS mount' % mp) - continue - if not since: print_mountstats(stats, nfs_only, rpc_only) elif since and mp not in old_mountstats:Signature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Re: [nfs-utils RFC PATCH 11/15] mountstats: Allow mountstats_command to take a variable number of mountpoints
Quoted 5 lines · Scott Mayhew
Quoted 3 lines · Chuck Lever
On Nov 5, 2014, at 12:01 PM, Scott Mayhew <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Allow the mountstats command to take a variable number of mountpoints (including none, in which case it will print stats for all NFS mountpoints it finds).
Cool idea.
Thanks! It looks like I need to add some sort of a header when only the rpc stats are being printed though.
You mean a header visually separating the group of stats for each mount point. Yeah, that would be helpful. (I typically have just one mount point under test on my system, so I didn’t notice this before).
Quoted 76 lines
Quoted 65 lines
Quoted 65 lines
Signed-off-by Scott Mayhew <[email protected]>---
tools/mountstats/mountstats.py| 32 +++++++++++++++++++-------------1 file changed, 19 insertions(+), 13 deletions(-)tools/mountstats/mountstats.py +19
-13@@ -616,10 +616,6 @@ def mountstats_command(): assert False, "unhandled option" mountpoints += args - if mountpoints == []: - print_mountstats_help(prog) - return - if rpc_only == True and nfs_only == True: print_mountstats_help(prog) return@@ -628,21 +624,31 @@ def mountstats_command(): infile = '/proc/self/mountstats' mountstats = parse_stats_file(infile) + # make certain devices contains only NFS mount points + if len(mountpoints) > 0: + check = [] + for device in mountpoints: + stats = DeviceData() + stats.parse_stats(mountstats[device]) + if stats.is_nfs_mountpoint(): + check += [device] + mountpoints = check + else: + for device, descr in mountstats.items(): + stats = DeviceData() + stats.parse_stats(descr) + if stats.is_nfs_mountpoint(): + mountpoints += [device] + if len(mountpoints) == 0: + print('No NFS mount points were found') + return + if since: old_mountstats = parse_stats_file(since) for mp in mountpoints: - if mp not in mountstats: - print('Statistics for mount point %s not found' % mp) - continue - stats = DeviceData() stats.parse_stats(mountstats[mp]) - - if not stats.is_nfs_mountpoint(): - print('Mount point %s exists but is not an NFS mount' % mp) - continue - if not since: print_mountstats(stats, nfs_only, rpc_only) elif since and mp not in old_mountstats:Signature · 1 line
1.9.3Quoted 11 lines
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
RFC PATCH 12/15 mountstats: Add support for -R/--raw to mountstats_command
In reply to Scott MayhewThis option displays the mountstats in raw format (i.e. in the same format as /proc/self/mountstats). It's intended to be used mainly with the -S/--since option.
--- tools/mountstats/mountstats.py | 50 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 5 deletions(-)
tools/mountstats/mountstats.py +45 -5
@@ -305,6 +305,41 @@ class DeviceData: return True return False + def display_raw_stats(self): + """Prints out stats in the same format as /proc/self/mountstats + """ + print('device %s mounted on %s with fstype %s %s' % \ + (self.__nfs_data['export'], self.__nfs_data['mountpoint'], \ + self.__nfs_data['fstype'], self.__nfs_data['statvers'])) + print('\topts:\t%s' % ','.join(self.__nfs_data['mountoptions'])) + print('\tage:\t%d' % self.__nfs_data['age']) + print('\tcaps:\t%s' % ','.join(self.__nfs_data['servercapabilities'])) + print('\tsec:\tflavor=%d,pseudoflavor=%d' % (self.__nfs_data['flavor'], \ + self.__nfs_data['pseudoflavor'])) + print('\tevents:\t%s' % " ".join([str(self.__nfs_data[key]) for key in NfsEventCounters])) + print('\tbytes:\t%s' % " ".join([str(self.__nfs_data[key]) for key in NfsByteCounters])) + print('\tRPC iostats version: %1.1f p/v: %s (nfs)' % (self.__rpc_data['statsvers'], \ + self.__rpc_data['programversion'])) + if self.__rpc_data['protocol'] == 'udp': + print('\txprt:\tudp %s' % " ".join([str(self.__rpc_data[key]) for key in XprtUdpCounters])) + elif self.__rpc_data['protocol'] == 'tcp': + print('\txprt:\ttcp %s' % " ".join([str(self.__rpc_data[key]) for key in XprtTcpCounters])) + elif self.__rpc_data['protocol'] == 'rdma': + print('\txprt:\trdma %s' % " ".join([str(self.__rpc_data[key]) for key in XprtRdmaCounters])) + else: + raise Exception('Unknown RPC transport protocol %s' % self.__rpc_data['protocol']) + print('\tper-op statistics') + prog, vers = self.__rpc_data['programversion'].split('/') + if vers == '3': + for op in Nfsv3ops: + print('\t%12s: %s' % (op, " ".join(str(x) for x in self.__rpc_data[op]))) + elif vers == '4': + for op in Nfsv4ops: + print('\t%12s: %s' % (op, " ".join(str(x) for x in self.__rpc_data[op]))) + else: + print('\tnot implemented for version %d' % vers) + print() + def display_nfs_options(self): """Pretty-print the NFS options """
@@ -565,7 +600,7 @@ def print_mountstats_help(name): print(' --since <file> shows difference between current stats and those in \'file\'') print() -def print_mountstats(stats, nfs_only, rpc_only): +def print_mountstats(stats, nfs_only, rpc_only, raw): if nfs_only: stats.display_nfs_options() stats.display_nfs_events()
@@ -573,6 +608,8 @@ def print_mountstats(stats, nfs_only, rpc_only): elif rpc_only: stats.display_rpc_generic_stats() stats.display_rpc_op_stats() + elif raw: + stats.display_raw_stats() else: stats.display_nfs_options() stats.display_nfs_bytes()
@@ -583,13 +620,14 @@ def mountstats_command(): """Mountstats command """ try: - opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since="]) + opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:R", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since=", "raw"]) except getopt.GetoptError as err: print_mountstats_help(prog) mountpoints = [] nfs_only = False rpc_only = False + raw = False infile = None since = None
@@ -612,6 +650,8 @@ def mountstats_command(): sys.exit(0) elif o in ("-S", "--since"): since = a + elif o in ("-R", "--raw"): + raw = True else: assert False, "unhandled option" mountpoints += args
@@ -650,14 +690,14 @@ def mountstats_command(): stats = DeviceData() stats.parse_stats(mountstats[mp]) if not since: - print_mountstats(stats, nfs_only, rpc_only) + print_mountstats(stats, nfs_only, rpc_only, raw) elif since and mp not in old_mountstats: - print_mountstats(stats, nfs_only, rpc_only) + print_mountstats(stats, nfs_only, rpc_only, raw) else: old_stats = DeviceData() old_stats.parse_stats(old_mountstats[mp]) diff_stats = stats.compare_iostats(old_stats) - print_mountstats(diff_stats, nfs_only, rpc_only) + print_mountstats(diff_stats, nfs_only, rpc_only, raw) def print_nfsstat_help(name): print('usage: %s [ options ]' % name)
Signature · 1 line
1.9.3
RFC PATCH 13/15 mountstats: Implement nfsstat_command
In reply to Scott MayhewDisplays nfssstat-like statistics for a single mountpoint (client statistics only).
--- tools/mountstats/mountstats.py | 87 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)
tools/mountstats/mountstats.py +86 -1
@@ -445,6 +445,42 @@ class DeviceData: print('\ttotal execute time: %f (milliseconds)' % \ (float(stats[7]) / count)) + def display_nfsstat_stats(self): + """Pretty-print nfsstat-style stats + """ + sends = 0 + trans = 0 + for op in self.__rpc_data['ops']: + sends += self.__rpc_data[op][0] + trans += self.__rpc_data[op][1] + retrans = trans - sends + print('Client rpc stats:') + print('calls retrans authrefrsh') + # authrefresh stats don't actually get captured in + # /proc/self/mountstats, so we fudge it here + print('%-11u%-11u%-11u' % (sends, retrans, sends)) + if not sends: + return + print() + prog, vers = self.__rpc_data['programversion'].split('/') + print('Client nfs v%d' % int(vers)) + info = [] + for op in self.__rpc_data['ops']: + print('%-13s' % str.lower(op), end='') + count = self.__rpc_data[op][0] + pct = (count * 100) / sends + info.append((count, pct)) + if (self.__rpc_data['ops'].index(op) + 1) % 6 == 0: + print() + for (count, pct) in info: + print('%-8u%3u%% ' % (count, pct), end='') + print() + info = [] + print() + for (count, pct) in info: + print('%-8u%3u%% ' % (count, pct), end='') + print() + def compare_iostats(self, old_stats): """Return the difference between two sets of stats """
@@ -706,9 +742,58 @@ def print_nfsstat_help(name): print() print(' nfsstat-like program that uses NFS client per-mount statistics.') print() + print('Options:') + print() + print(' -m <mountpoint>, --mountpount <mountpoint>') + print(' Show stats for \'mountpoint\'') + print() + print(' -f <file>, --file <file>') + print(' Read stats from \'file\' instead of /proc/self/mountstats') + print() + print(' -S <file>, --since <file>') + print(' Shows difference between current stats and those in \'file\'') + print() + print(' -h, --help') + print(' What you just did') def nfsstat_command(): - print_nfsstat_help(prog) + """nfsstat-like command for NFS mount points + """ + try: + opts, args = getopt.getopt(sys.argv[1:], "f:hm:S:", ["file=", "help", "mountpoint=", "since="]) + except getopt.GetoptError as err: + print_nfsstat_help(prog) + infile = None + mp = None + since = None + for o, a in opts: + if o in ("-f", "--file"): + infile = a + elif o in ("-h", "--help"): + print_nfsstat_help(prog) + sys.exit() + elif o in ("-m", "--mountpoint"): + mp = a + elif o in ("-S", "--since"): + since = a + else: + assert False, "unhandled option" + if not mp: + print_nfsstat_help(prog) + sys.exit() + if not infile: + infile = '/proc/self/mountstats' + mountstats = parse_stats_file(infile) + stats = DeviceData() + stats.parse_stats(mountstats[mp]) + if not since: + stats.display_nfsstat_stats() + else: + old_mountstats = parse_stats_file(since) + oldstats = DeviceData() + oldstats.parse_stats(old_mountstats[mp]) + diff_stats = stats.compare_iostats(oldstats) + diff_stats.display_nfsstat_stats() def print_iostat_help(name): print('usage: %s [ <interval> [ <count> ] ] [ <mount point> ] ' % name)
Signature · 1 line
1.9.3
RFC PATCH 14/15 mountstats: Remove the --start and --end options
In reply to Scott MayhewAll they do is throw 'not implemented' exceptions anyway.
--- tools/mountstats/mountstats.py | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
tools/mountstats/mountstats.py +2 -8
@@ -631,8 +631,6 @@ def print_mountstats_help(name): print(' --file <file> read stats from \'file\' instead of /proc/self/mountstats') print(' --nfs display only the NFS statistics') print(' --rpc display only the RPC statistics') - print(' --start sample and save statistics') - print(' --end resample statistics and compare them with saved') print(' --since <file> shows difference between current stats and those in \'file\'') print()
@@ -656,7 +654,7 @@ def mountstats_command(): """Mountstats command """ try: - opts, args = getopt.getopt(sys.argv[1:], "ef:hnrsvS:R", ["end", "file=", "help", "nfs", "rpc", "start", "version", "since=", "raw"]) + opts, args = getopt.getopt(sys.argv[1:], "f:hnrvS:R", ["file=", "help", "nfs", "rpc", "version", "since=", "raw"]) except getopt.GetoptError as err: print_mountstats_help(prog)
@@ -668,9 +666,7 @@ def mountstats_command(): since = None for o, a in opts: - if o in ("-e", "--end"): - raise Exception('Sampling is not yet implemented') - elif o in ("-f", "--file"): + if o in ("-f", "--file"): infile = a elif o in ("-h", "--help"): print_mountstats_help(prog)
@@ -679,8 +675,6 @@ def mountstats_command(): nfs_only = True elif o in ("-r", "--rpc"): rpc_only = True - elif o in ("-s", "--start"): - raise Exception('Sampling is not yet implemented') elif o in ("-v", "--version"): print('%s version %s' % (sys.argv[0], Mountstats_version)) sys.exit(0)
Signature · 1 line
1.9.3
RFC PATCH 15/15 mountstats: Update the help output
In reply to Scott MayhewUpdated the help output for mounstats, ms-iostat, and ms-nfsstat to make them constistent.
--- tools/mountstats/mountstats.py | 43 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 6 deletions(-)
tools/mountstats/mountstats.py +37 -6
@@ -627,11 +627,25 @@ def print_mountstats_help(name): print() print(' Display NFS client per-mount statistics.') print() - print(' --version display the version of this command') - print(' --file <file> read stats from \'file\' instead of /proc/self/mountstats') - print(' --nfs display only the NFS statistics') - print(' --rpc display only the RPC statistics') - print(' --since <file> shows difference between current stats and those in \'file\'') + print('Options:') + print() + print(' -n, --nfs') + print(' Display only the NFS statistics') + print() + print(' -r, --rpc') + print(' Display only the RPC statistics') + print() + print(' -f <file>, --file <file>') + print(' Read stats from \'file\' instead of /proc/self/mountstats') + print() + print(' -S <file>, --since <file>') + print(' Shows difference between current stats and those in \'file\'') + print() + print(' -h, --help') + print(' Show the help message') + print() + print(' -v, --version') + print(' Display the version of this command') print() def print_mountstats(stats, nfs_only, rpc_only, raw):
@@ -748,7 +762,8 @@ def print_nfsstat_help(name): print(' Shows difference between current stats and those in \'file\'') print() print(' -h, --help') - print(' What you just did') + print(' Show the help message') + print() def nfsstat_command(): """nfsstat-like command for NFS mount points
@@ -810,6 +825,22 @@ def print_iostat_help(name): print(' mount points will be displayed. Otherwise, all NFS mount points on the') print(' client are listed.') print() + print('Options:') + print() + print(' -f <file>, --file <file>') + print(' Read stats from \'file\' instead of /proc/self/mountstats.') + print(' This may not be combined with the <interval> and <count> parameters.') + print() + print(' -S <file>, --since <file>') + print(' Shows difference between current stats and those in \'file\'') + print(' This may not be combined with the <interval> and <count> parameters.') + print() + print(' -h, --help') + print(' Show the help message') + print() + print(' -v, --version') + print(' Display the version of this command') + print() def print_iostat_summary(old, new, devices, time): for device in devices:
Signature · 1 line
1.9.3
Hi Scott-
Quoted 19 lines · Scott Mayhew
The following patches add a couple of enhancements to
mountstats.py. I also fixed a few bugs I encountered along the way. Highlights include:- added support for -f/--file to allow stats to be parsed from an aritrary input file instead of
/proc/self/mountstats- added support for -S/--since to show just the changes that have occurred between the current and a previous set of statisics (works with and without the -f option)
- added support for -R/--raw to generate 'raw' statistics (i.e. in the same format as
/proc/self/mountstats). It's intended to be used with the -f and -S options.- implemented the ms-nfsstat command to generate client-side nfsstat-like statisics (only works with a single mountpoint)
Quoted 8 lines
My motivation for these changes was so that I could take various copies of
/proc/self/mountstatsand massage them into data that I could feed into the 'report' option of Dros's nfsometer tool for scenarios where it's not feasible to run nfsometer itself (e.g. systems where we can't start with an 'idle' state (i.e. no NFS filesystems initially mounted), systems with multiple NFS filesystems mounted, and workloads that can't easily be boiled down into an nfsometer workload file or run via the custom workload environment variables).
Do you have a git repo I can pull these from to try out?
Quoted 31 lines
Scott Mayhew (15):
mountstats: Fix up NFS event counters
mountstats: Add lists of various counters
mountstats: Refactor__parse_nfs_lineand__parse_rpc_line
mountstats: Refactorcompare_iostats
mountstats: Convert existing option parsing to use the getopt module
mountstats: Make ms-iostat output match that ofnfs-iostat.py
mountstats: Makeprint_iostat_summaryhandle newly appearing mounts
mountstats: Add support for -f/--file to the mountstats and ms-iostat
commands
mountstats: Add support for -S/--since to the mountstats and ms-iostat
commands
mountstats: Fix IndexError in__parse_nfs_line
mountstats: Allowmountstats_commandto take a variable number of
mountpoints
mountstats: Add support for -R/--raw tomountstats_command
mountstats: Implementnfsstat_command
mountstats: Remove the --start and --end options
mountstats: Update the help output
tools/mountstats/mountstats.py| 730 ++++++++++++++++++++++++++++++-----------1 file changed, 544 insertions(+), 186 deletions(-)mode change 100644 => 100755
tools/mountstats/mountstats.pySignature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Chuck,
Quoted 30 lines · Chuck Lever
Hi Scott-
Quoted 19 lines · Scott Mayhew
The following patches add a couple of enhancements to
mountstats.py. I also fixed a few bugs I encountered along the way. Highlights include:- added support for -f/--file to allow stats to be parsed from an aritrary input file instead of
/proc/self/mountstats- added support for -S/--since to show just the changes that have occurred between the current and a previous set of statisics (works with and without the -f option)
- added support for -R/--raw to generate 'raw' statistics (i.e. in the same format as
/proc/self/mountstats). It's intended to be used with the -f and -S options.- implemented the ms-nfsstat command to generate client-side nfsstat-like statisics (only works with a single mountpoint)
Quoted 8 lines
My motivation for these changes was so that I could take various copies of
/proc/self/mountstatsand massage them into data that I could feed into the 'report' option of Dros's nfsometer tool for scenarios where it's not feasible to run nfsometer itself (e.g. systems where we can't start with an 'idle' state (i.e. no NFS filesystems initially mounted), systems with multiple NFS filesystems mounted, and workloads that can't easily be boiled down into an nfsometer workload file or run via the custom workload environment variables).
Quoted 1 line
Do you have a git repo I can pull these from to try out?
You can pull them from https://github.com/scottmayhew/nfs-utils.git
commit ids f9c256040^..6a43a1599
-Scott
Quoted 33 lines
Quoted 33 lines
Scott Mayhew (15):
mountstats: Fix up NFS event counters
mountstats: Add lists of various counters
mountstats: Refactor__parse_nfs_lineand__parse_rpc_line
mountstats: Refactorcompare_iostats
mountstats: Convert existing option parsing to use the getopt module
mountstats: Make ms-iostat output match that ofnfs-iostat.py
mountstats: Makeprint_iostat_summaryhandle newly appearing mounts
mountstats: Add support for -f/--file to the mountstats and ms-iostat
commands
mountstats: Add support for -S/--since to the mountstats and ms-iostat
commands
mountstats: Fix IndexError in__parse_nfs_line
mountstats: Allowmountstats_commandto take a variable number of
mountpoints
mountstats: Add support for -R/--raw tomountstats_command
mountstats: Implementnfsstat_command
mountstats: Remove the --start and --end options
mountstats: Update the help output
tools/mountstats/mountstats.py| 730 ++++++++++++++++++++++++++++++-----------1 file changed, 544 insertions(+), 186 deletions(-)mode change 100644 => 100755
tools/mountstats/mountstats.pySignature · 6 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Scott Mayhew
The following patches add a couple of enhancements to
mountstats.py. I also fixed a few bugs I encountered along the way. Highlights include:- added support for -f/--file to allow stats to be parsed from an aritrary input file instead of
/proc/self/mountstats
Quoted 7 lines
- added support for -S/--since to show just the changes that have occurred between the current and a previous set of statisics (works with and without the -f option)
- added support for -R/--raw to generate 'raw' statistics (i.e. in the same format as
/proc/self/mountstats). It's intended to be used with the -f and -S options.
The mountstats(8) will need to be updated with these new flags... You could probably just use the verbiage from the usage messages
Ping me if want some help with the archaic nroff notation ;-)
Quoted 4 lines
- implemented the ms-nfsstat command to generate client-side nfsstat-like statisics (only works with a single mountpoint)
Quoted 8 lines
My motivation for these changes was so that I could take various copies of
/proc/self/mountstatsand massage them into data that I could feed into the 'report' option of Dros's nfsometer tool for scenarios where it's not feasible to run nfsometer itself (e.g. systems where we can't start with an 'idle' state (i.e. no NFS filesystems initially mounted), systems with multiple NFS filesystems mounted, and workloads that can't easily be boiled down into an nfsometer workload file or run via the custom workload environment variables).
Thanks for doing this!
steved.
Quoted 25 lines
Scott Mayhew (15):
mountstats: Fix up NFS event counters
mountstats: Add lists of various counters
mountstats: Refactor__parse_nfs_lineand__parse_rpc_line
mountstats: Refactorcompare_iostats
mountstats: Convert existing option parsing to use the getopt module
mountstats: Make ms-iostat output match that ofnfs-iostat.py
mountstats: Makeprint_iostat_summaryhandle newly appearing mounts
mountstats: Add support for -f/--file to the mountstats and ms-iostat
commands
mountstats: Add support for -S/--since to the mountstats and ms-iostat
commands
mountstats: Fix IndexError in__parse_nfs_line
mountstats: Allowmountstats_commandto take a variable number of
mountpoints
mountstats: Add support for -R/--raw tomountstats_command
mountstats: Implementnfsstat_command
mountstats: Remove the --start and --end options
mountstats: Update the help outputtools/mountstats/mountstats.py | 730 ++++++++++++++++++++++++++++++----------- 1 file changed, 544 insertions(+), 186 deletions(-) mode change 100644 => 100755 tools/mountstats/mountstats.py