Performance and Statistics Collector (Perfstat 7-mode) 7.39 UNIX/Linux, 7.39 Windows - Description Note: NetApp offers limited support for this tool. Please use the feedback form to request support. Last updated: 25-April-2013 Note: Please always verify that you are using the most recent version of Perfstat. Older versions may trigger bugs which degrade filer performance or stability. Overview Perfstat is a data collection tool with several key properties: Captures all needed performance information with one command Captures information from host(s) and filer(s) Captures all information simultaneously for cross correlation Operates on all host platforms and all filer platforms All captured data is recorded in a single plain-text output file Perfstat 7 comes in exactly two flavors: Unix/Linux version (perfstat.sh) Windows version (perfstat.exe) Supported platforms: Unix: AIX, ESX, HP-UX, Linux, OSF1, Solaris, FreeBSD Windows: 2000/XP/2003/2008*/7* *NOTE: Windows 2008 and Windows 7 support requires openSSHforWindows to be setup. Please follow the procedure available in "openSSH_procedure.txt" file bundled with perfstat7 windows binary to achieve the same. Documentation Running perfstat without any flags will provide basic usage instructions. Unix Perfstat: Version 7.39 4-2013 - perfstat.sh is a simple Bourne Shell script that captures performance and configuration statistics. - Output from perfstat is sent to standard out and is typically captured in an output file for later analysis. - perfstat.sh is capable of capturing info from host(s) and NetApp storage controllers simultaneously. - Currently perfstat supports these OS platforms: Solaris, HP-UX, OSF1, Linux, AIX, FreeBSD, OpenBSD, ESX3.5 - perfstat.sh is typically run as root from the host or as a user with root-level permissions - For controller data capture, the user must have RSH or SSH privileges to the controller. Unless instructed otherwise, perfstat will use 'root' as the default username to communicate remotely with storage controllers and hosts. Usage: (basic options list) perfstat [-f controllername] [-t time] > perfstat.out where: -f controllername - host name (or IP address) of target controller -t time - collect performance data for 'time' minutes Simple Examples: Capture data on local host and one controller for 5 minutes: perfstat -f controller1 -t 5 > perfstat.out Capture data on multiple hosts and controllers for 10 minutes: perfstat -h host1,host2 -f controller1,controller2 -t 10 > perfstat.out Capture data for five 1 minute iterations, with 10 minutes between successive iterations: perfstat -f controller1 -t 1 -i 5,10 > perfstat.out Usage: (more complete options list) perfstat [-f controllername[,controllername1,controllername2,...]] [-h hostname[,hostname1,hostname2,...]] [-t time] (sample time per iteration, default 2) [-i n[,m]] (repeat n times with m minutes between samples, defaults: n=1, m=0) [-I] (force perfstat to execute all iterations) [-r rootcmd] (e.g. sudo) [-l login[:password]] (RSH/SSH login and RSH password) [-S] (use SSH instead of RSH) [-s param1[,value1][:param2[,value2]]...] (optional RSH/SSH arguments) [-F] (do not capture information from local host) [-V] (do not capture vfiler data) [-p] (capture performance data only, no config info) [-c] (capture config info only, no performance data) [-L] (capture logs - beware verbose output) [-E cmd[,cmd2,cmd3] (exclude commands) [-P domain1[,domain2,domain3...] (capture profiles, use "-P flat" to capture complete profile) [-a app_name -o app_param] (E.g., -a oracle) [-v] (print version info only) [-q] (quiet mode - suppress all console output) [-x] (print what commands will be issued without actually issuing them) [-d] (debug mode - beware verbose output) [-b] (begin sampling and return immediately) [-e] (end sampling - used in conjunction with -b) [-n] (RAM Service Invocation) [-k] (disable collection of "stutter" statit; i.e., collect 1 statit report that covers the entire iteration) [-K] (collect only "stutter" statit reports over the entire iteration) [-T default | sk_mod,level[,sk_mod2,level,...]] (collect sktrace) [-B sk_buffer_size] (specify sktrace buffer size) [--cluster] (toggles C-mode data collection) [-w] (total number of minutes to wait for perfstat execution to complete. Default would be 5 min.) [-y] (force perfstat to execute only critical set of commands.) [--wafl-bufstats] (force perfstat to execute "wafl bufstats" commands, by default it does not execute.) Notes: -h option adds hosts to be monitored. By default, the local host is always monitored, unless the -F flag is specified. E.g., executing this command perfstat -h host1 > perfstat.out on machine host0 will result in data captured from both host0 and host1 This command: perfstat -F -h host1 > perfstat.out on machine host0 will result in data captured from host1 only -l option is only applied to RSH/SSH commands to the controller. RSH/SSH commands to other hosts do not use the -l information. perfstat.sh does not support password authentication over SSH, so if '-S -l login:password' is specified, the password will be stripped from the subsequent SSH invocations. -S requires passwordless (public key) SSH authentication to be configured for all controllers and hosts An SSH username may prefix a hostname using a '@' E.g., perfstat -S -h root@host1,user@host2 Additionally, the -l option can be used to specify usernames for controller login. E.g., perfstat -S -f controllername -l user -s arguments to this option use the syntax 'param,value', and param value pairs can be separated by a ':' E.g. the syntax '-S -s p1,22:p2:p3,v3' tells perfstat to build the following SSH invocation string: 'SSH -o BatchMode=yes -2 -ax -p1 22 -p2 -p3 v3' -a is limited to these applications currently: oracle: -o specifies sub arguments. run -o help for details -b|-e are provided for legacy compatibility and can be used to manually perform an iteration. Use -b to start the iteration and after perfstat returns, use -e to stop that iteration. -P saves profiling data in a tar.gz file in the current working directory and deletes any existing gmon files on the controller -E excludes all foreground commands that have at least the cmd as a substring; E.g. -E snap,vol - excludes all 'snap*' and 'vol*' commands -E "snap list -v" - excludes only the command 'snap list -v' -T used to toggle sktrace collection and specify sk modules and the levels at which trace data should be collected for those modules. Some valid sk modules include SK, WAFL, DISK, and SCSITARGET. For default values of 'SK 7 WAFL 4', specify '-T default'. sktrace data will be copied off of the controller(s) and into the current working directory from where perfstat was invoked. -B used to manually set the sktrace buffer size. By default, perfstat will use a default value of '40m' (40MB). Please note that it is required that the units be specified with the size in the format: k (KB), m (MB), or g (GB). This value should only be changed if specifically advised by global support. -y option must be used only in critical situations when filer does not respond. If this option is enabled, host data will not be collected It is advised to use this option with default iteration time and no profile and sktrace collection should be done. Early termination of execution: as of v7.00 perfstat will terminate iterations early if a calculated max runtime is met or exceeded. If it is required that perfstat must execute all iterations regardless of the total runtime, please use the '-I' option. Windows Perfstat: Version 7.39 10-2013 - perfstat.exe is a tool for capturing performance and configuration statistics. - Output from perfstat is sent to standard out and is typically captured in an output file for later analysis. - perfstat.exe is capable of capturing info from one Windows host and multiple NetApp storage controllers simultaneously. - For controller data capture, the user must have RSH or SSH privileges to the controller. Unless instructed otherwise, perfstat will use 'root' as the default username to communicate remotely with storage controllers. Usage: (basic options list) perfstat [-f controllername] [-t time] > perfstat.out where: -f controllername - host name or IP address of target controller -t time - collect performance data for 'time' minutes Simple Example: Capture data on local host and one controller for 5 minutes: perfstat -f controller1 -t 5 > perfstat.out Capture data on multiple hosts and controllers for 10 minutes: perfstat -f controller1,controller2 -t 10 > perfstat.out Capture data for five 1 minute iterations, with 10 minutes between successive iterations: perfstat -f controller1 -t 1 -i 5,10 > perfstat.out Usage: (more complete options list) perfstat [-f controllername[,controllername1,controllername2,...]] [-t time] (sample time per iteration, default 2) [-i n[,m]] (repeat n times with m minutes between samples, defaults: n=1,m=0) [-I] (force perfstat to execute all iterations) [-l login[:password]] (rsh/ssh login and password for rsh only) [-F] (do not capture information from local host) [-V] (do not capture vfiler data) [-p] (capture performance data only, no config info [-c] (capture config info only, no performance data [-L] (capture logs - beware verbose output [-E cmd[,cmd2,cmd3]... (exclude commands) [-P domain1[,domain2,domain3...] (capture profiles, use "-P flat" to capture complete profile) [-v] (print version info only) [-q] (quiet mode - suppress all console input) [-x] (print what commands will be issued without actually issuing them [-d] (debug mode - beware verbose output) [-m msinfo_path] (path to msinfo32.exe) [-S pw:|kf:] (use ssh instead of rsh) [-k] (disable collection of "stutter" statit; i.e., collect 1 statit report that covers the entire iteration) [-K] (collect only "stutter" statit reports over the entire iteration) [-T default | sk_mod,level[,sk_mod2,level,...]] (collect sktrace) [-B sk_buffer_size] (specify sktrace buffer size) [-C seconds] (the number of seconds to wait on command processes- default is 70 seconds) [-w minutes] (total number of minutes to wait for perfstat execution to complete. Default will be 5 minutes) [-g] (force perfstat to use ssh.exe instead of plink.exe) [-y] (force perfstat to execute only critical set of commands) [--wafl-bufstats] (force perfstat to execute "wafl bufstats" commands, by default it does not execute.) Notes: -P saves profiling data in a subdirectory 'gmon' of the current working directory and deletes any existing gmon files on the controller -E exludes all foreground commands that have at least the cmd as a substring; E.g. -E snap,vol - excludes all 'snap*' and 'vol*' commands -E "snap list -v" - excludes only the command 'snap list -v' -S requires the plink ssh client to be available in the working path To authenticate with a password, use the syntax -S pw: For public key authentication, use the syntax -S kf:, where keyfile is a Putty private keyfile created with puttygen. plink and puttygen are available for free download at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Additionally, the -l option can be used to specify usernames for controller login. E.g., perfstat.exe -S -l user -T used to toggle sktrace collection and specify sk modules and the levels at which trace data should be collected for those modules. Some valid sk modules include SK, WAFL, DISK, and SCSITARGET. For default values of 'SK 7 WAFL 4', specify '-T default'. sktrace data will be copied off of the controller(s) and into the current working directory from where perfstat was invoked. values of 'SK 7 WAFL 4', specify '-T default'. sktrace data will be copied off of the controller(s) and into the current working directory from where perfstat was invoked. -B used to manually set the sktrace buffer size. By default, perfstat will use a default value of '40m' (40MB). Please note that it is required that the units be specified with the size in the format: k (KB), m (MB), or g (GB). This value should only be changed if specifically advised by global support. -g supports only public key authentication. The key must be provided with "-S" option using the syntax -S kf:, where keyfile is the openSSH private keyfile generated by ssh-keygen executable. -y option must be used only in critical situations when filer does not respond. If this option is enabled, host data will not be collected It is advised to use this option with default iteration time and no profile and sktrace collection should be done. Early termination of execution: as of v7.00 perfstat will terminate iterations early if a calculated max runtime is met or exceeded. If it is required that perfstat must execute all iterations regardless of the total runtime, please use the '-I' option. Monitoring for 24 hour periods A common practice is to capture perfstat data 24 hours a day and archive some number of data samples (e.g. 1 week worth). The recommended method for doing this is to capture multiple small samples throughout the day, rather than one 24 hour sample. Many customers script this or place it in the cron job log. Here are some typical examples: perfstat -f filer1 -t 30 -i 46 > perfstat.$date.out NOTE: For Windows users the example below shows a nice way to format perfstat output file names with date and time information: perfstat -f filer1 -t 30 -i 46 > perfstat-"Date_%date:~4,2%%date:~7,2%%date:~10,4%-Time_%time:~0,2%h%time:~3,2%m%time:~6,2%s" Both examples above will capture 46 samples of 30 minute intervals. This is a total of 23 hours. Since perfstat uses a lot of rsh commands and the time to complete the rsh's can vary, it often runs for slightly longer than the specified time. To compensate for this (and not have 2 perfstat tools running concurrently), we recommend a 23 hour run to cover a 24 hour period. Download