							Converged Perfstat Release Guide:
							--------------------------------

1.	Introduction:
	-------------

Perfstat Converged is a diagnostics data collection tool for Data ONTAP running in Cluster-Mode and Data ONTAP in running 7-mode, with several key properties:
	*	Captures all needed performance information with one command.
	*	Captures information from node(s).
	*	Captures information from local host and remote Host(s).
	*	Captures information from network switch(s).
	*	Captures all information simultaneously for cross correlation
	*	All captured data is collated into a single zip/tar file.

Converged Perfstat comes in exactly three flavors:
	*	Linux version (perfstat8)
	*	Windows version (perfstat8.exe)
	*	Apple Mac OS X (perfstat8)

2.	Supported platforms.
	--------------------
	Converged Perfstat8 supports various Data ONTAP versions, local hosts, remote hosts and switches.

	2.1	Supported Data ONTAP versions

	*	ONTAP 8.0.x (C-mode)
	*	ONTAP 8.1+ (7-mode)
	*	ONTAP 8.1+ (C-mode)

	2.2	Supported Client OS versions

	*	Windows 2003
	*	Windows 2008
	*	Windows XP 
	*	Windows Vista
	*	Windows 7
	*	Linux
	*	Apple Mac OS X Snow Leopard [ Intel x86-64 ]
	*	Apple Mac OS X Lion [ Intel x86-64 ]

	2.3	Supported Switches:

	Converged perfstat is capable of collecting data from network switches via SSH. 
	The pre-requisite to collect data from the switch is that the switch must support SSH and SSH authentication and network communication must be enabled. 
	The currently supported switch vendors/models are:
	*	Cisco: cisco nexus 5000 series
	*	Brocade: brocade 4100 switch series
	*	Qlogic: SANbox 5602 FC switch

	2.4	Supported Local host OS versions:

	Perfstat Converged collects data from local host by default. The following Operating systems are supported for local host.

		1.	Linux

		2.	Windows

			*	Windows 7
			*	Windows XP
			*	Windows Vista
			*	Windows 2003
			*	Windows 2008
		3.	Apple Mac OS X

			*	Mac OS X Snow Leopard [ Intel x86-64 ]
			*	Mac OS X Lion [ Intel x86-64 ]
			
	2.5	Supported Remote host OS versions:

		Converged Perfstat collects data from the remote host(s) if specified with command line. The prerequisite to collect data from remote hosts
		is that it should have support for SSH. The only exception to this rule is for hosts running AIX where RSH is mandatorily configured. The following Operating systems
		are supported as remote host:

		*	Linux
		*	Solaris
		*	HP-UX
		*	FreeBSD
		*	ESX server 3.5
		*	AIX


3.	Converged Perfstat 7-mode:
	--------------------------

	Converged Perfstat in current mode executes by establishing an interactive session with the node. There is a limitation with 7-Mode, that only one SSH/Telnet interactive 
	session could be active at any instance of time. Because of this constraint if any session is opened with the node, Converged Perfstat fails to execute in current mode for 
	nodes running data ONTAP 7-mode. To overcome this limitation in 7-Mode, a non-interactive way of  connectivity with nodes, is been added. This newly added capability is a 
	called "Legacy Mode". With this capability, Converged Perfstat 7-mode executes alike perfstat7. In Legacy Mode, as there is no interactive session established with the node
	so systemshell and SKTrace data will not be collected in Legacy Mode. To achieve this Legacy Mode capability, a new CLI option [ LEGACY_MODE ] is been added in Converged 	Perfstat.

	To summarize, Converged Perfstat 7-mode can be executed in:
		
		*	Current Mode
		*	Legacy Mode   [ Using --LEGACY_MODE CLI option ]
		*	Starting with Current Mode and Later dynamically switching to LEGACY MODE due to any active SSH/Telnet session.

4.	Pre-requisite to run Converged Perfstat in 7-Mode:
	-------------------------------------------------

	There are set of rules which need to be followed while using Converged Perfstat to collect data from the controllers running with Data ONTAP 7-Mode :

	*	Use puttygen.exe generated public/private SSH key pair for passwordless perfstat run on windows client .
	*	On Windows clients that are used to execute Perfstat in 7-Mode, the file plink.exe must be present in the perfstat 
		run directory to run parallel commands if the SSH option is enabled at command line.
	*	The controller must have been added to the 'trusted known host list' on the client before running perfstat. For more details, Please see the FAQ section.
	*	While running Converged Perfstat 7-mode with 'RSH' [ i.e. default protocal ] either in current mode or Legacy Mode, the "/etc/hosts.equiv" file
		must have been appended already with the "<hostname>  <username>". For the Apple Mac OS X hosts, rsh need to be enabled on the host first.For more details, 
		please see the FAQ section.
	*	The 'diag' user account must be unlocked on the controller to collect SystemShell data from it.
	*	While running Converged Perfstat 7-mode in Legacy mode on Windows2008/Windows7 systems, openSSH must have been setup already by following the procedure given
           in  "openssh_procedure.txt" bundled with the perfstat binary
	


	4.1	Prerequisite to run parallel commands in 7-Mode in current mode:

	When Converged Perfstat 7-Mode is run in current mode, the parallel commands will be run either through RSH or SSH.
	The default setting for parallel command execution is to use RSH. The prerequisite to run parallel commands using RSH is the local host details should be 
	appended into  "/etc/hosts.equiv" on the controller running Data ONTAP 7-Mode. The same could be achieved  by executing the below command in the controller shell
	
	Example:
	wrfile -a /etc/hosts.equiv "host_ip login_name"

	If SSH is to be used for parallel commands and the client running Perfsstat is Windows, then the plink.exe file must be in the same directory as the perfstat binary.


5.	Converged Perfstat C-mode:
	--------------------------

	Converged Perfstat C-mode executes only in current mode by establishing the interactive SSH session with the node. The Legacy mode is not supported in C-mode.

6.	Pre-requisite to run Converged Perfstat in C-mode:
	--------------------------------------------------

	There are set of rules which need to be followed while using Converged Perfstat to collect data from the controllers running with Data ONTAP C-mode.

	*	The 'diag' user account must be unlocked on the controller to collect SystemShell data.
	*	On Windows clients that are used to execute Perfstat in cluster-mode, the file plink.exe must be present in the perfstat
		run directory to download perfstat generated files from the controllers onto the host.






7.	Caveats & Known Issues:
	----------------------

	a.	Perfstat8 may fail or behave unexpectedly if:
		
		*	A controller name cannot be resolved from the host running the perfstat utility.
		*	User does not have write permissions in the directory where perfstat is running.
		*	A system shell account for the "diag" user is locked
		*	multiple perfstat instances are run simultaneously from within the same directory.
		*	debug messages are directed to console on the Data ONTAP 7-Mode node.

	
	 
  

	b.	When using Windows clients to run Perfstat, when specifying the password as part of the Perfstat authentication, if the password 
		has an "@" character, place quotes at the beginning and end of the password. 

	c.	The "--nodes" option is not supported for collected performance information from multiple nodes running in different Data ONTAP Modes 
		(7-Mode/Cluster-Mode).  If performance data is needed from Data ONTAP operating in 7-Mode and Data ONTAP operating in Cluster-Mode nodes at the same 
		time, two separate instances of Perfstat converged should be used, one for each Mode.

	d.	Perfstat running on Linux clients only supports empty passphrases.

	e.	Known Burts
 
		BURT	Description	   
		
		405679	Multiple instances of perfstat running against the same cluster will cause one instance of perfstat to fail.	   
		316710	Requests can time out if a controller is under severe load, resulting in all data not being collected.	   
		391050	Linux Perfstat can fail on a 24 node cluster.	   
		574443	In 7-mode, if the system shell password is provided wrong at the first time, Converged perfstat hangs during the second try.
		

8.	Structure of the output file:

	Output file-
		*	console.log
		*	Perfstat.preset
				*	node|	 	
						1.	node-ip1	
									*	output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
	
						2.	node-ip2	
									*	 output.data
									*	output.index
									*	Logs (if rastrace (or) profile (or) sktrace (or) log is used).
							...		
				*	host		
						1.	Local-host-IP	
									*	output.data
									*	output.index
						2.	Remote-host-ip1	
									*	output.data
									*	output.index
						3.	Remote-host-ip2	
									*	output.data
									*	output.index
						        .
				*	switch
						1.	switch-ip1	
									*	output.data
									*	output.index
						2.	switch-ip2 
									*	output.data
									*	output.index
							..


9.	Help Manual:
	------------

	NAME 
	perfstat v8.1 (20121020_1826517) Supported Converged ONTAP

	SYNOPSIS
	Perfstat is a performance diagnostic tool supplied and supported by NetApp.
	Usage:  perfstat [OPTIONS] <a node-management IP of a node in the cluster>

	OPTIONS:
	-h      --help                To see a complete detailed usage
	-v      --version             Prints perfstat version, and
								  supported ONTAP versions.
	-q      --quiet               Suppress all non-error messages
			--verbose             Increase verbosity 
								(beware: prints all command executions)
			--debug               Beware verbose output. Both 'verbose'
								and 'debug' will add client-side timestamps.
			--preset-file         Use a different preset file.
			--print-preset        Prints the internal preset file.
	-o      --output-file         Use a different output file name.
								(default: "perfstat_data_yyyymmdd_hhmmss.zip").
			--sshprivatekey-file  SSH private key filename for admin user.
			--sample-interval     Specifies statit/sysstat frequency.
			--config-all          captures config information in every iteration.
								(default: CONFIG info is captured only during 
	the first iteration).
	-t      --time                sample time in minutes, per iteration.
								(default: 2 minutes).
	-i      --iteration=N[,M]     repeat N times with M seconds between samples.
								(default: N=1, M=0)..
	-n      --nodes=N1[,N2,N3,..] node names to collect from.
								(defaults to all nodes).
			--max-run-time        Maximum Perfstat runtime in minutes.
			--full-stutter-statit Split entire time (-t)
								into all stutter statit collections.
			--no-stutter-statit   Disable 'stuttered' statit commands.
			--enable-flag=PATTERN Enables data collection for specified flags.
								(default: All flags are disabled.
								Command with !<flag-name> flag value will be executed).
			--unzip=FILE          Unzips the specified file.
			--include=PATTERN     Includes *only* commands matching 
								PATTERN, and excludes everything else.
			--exclude=PATTERN     Excludes commands matching PATTERN,
								and includes everything else.
	-H      --hosts=H1:username:passWord[,H2:userName:passWord,H3:passWord,..]Remote host names to collect from.
	-m      --mode=[7-mode/cluster-mode/cluster/c]	Specify the mode for the target filers (default: 7-mode).
	-z      --FILER_ONLY           Specify the flag  to capture information from filer(s) only (default: Host and Filer(s)
	-s      --SSH                 How the 7-mode parallel commands should be executed, using SSH or RSH. (default: RSH)
	-r      --MODULE_INST =ID1[ID2,ID3..]Capture RAS trace for the instances of pre-configured modules(default: instance-id 1)
	-p      --PROFILES=[kahuna,flat,storage,(more)]Capture MP Domain profiles from filers
	-l      --login               Filer login user in 7-mode [ default: root ]
	-a      --app_name            Name of the application for which data to be collected on host side.
	-w      --app_args            List of parameters for the application data collection on host side. 
	-S      --switch              Collect the data from target switches.The format should <SwitchIP>:<SwitchType>:<loginUser>:<loginPassword>
	-k      --SKTRACE_POINT       Collect the sktrace for the modules. The format should be <Module>:<debug level> (deafult: 'SK:7,WAFL:4'). The debug level should be in old data ONTAP7.x fashion
	-L      --log                 Collect back up log information
			--LEGACY_MODE         Collects output in Legacy Mode. This option work only for 7-mode. In Legacy mode, Systemshell and SKTrace data will not be collected.
			--diag-passwd         diag (systemshell) password. Format should be <systemShellPassword> for C-Mode. <IP1>:<systemShellPassword1>,<IP2>:<systemShellPassword2> for 7-Mode
	
	OPTION DESCRIPTION
	1. --include=PATTERN/--exclude=PATTERN
	The include/exclude patterns consist of semicolon separated parameter tags, which match those from the preset file.
	The parameter tags themselves have comma separated values. Any definition lines in the preset file which match ANY
	parameter tag and value will be included or excluded from execution. include/exclude option may not be used together.
	Here are a few examples:
	
	a. --exclude="SHELL=SYSTEMSHELL"
	With this option perfstat will not execute any systemshell commands.
	b. --include="TYPE=PERF"
	With this option perfstat will only execute commands with TYPE as "PERF".
	
	2. --enable-flag=PATTERN
	The enable-flag pattern consist of comma separated flag names, which match those from the preset file.
	Any definition lines in the preset file which match ANY flag value will be enabled and added for execution.
	Here are a few examples:
	
	a. --enable-flag="snapmirror,wafl" With this option perfstat will enable "snapmirror" and "wafl" flags. Command with "snappmirror" and
	"wafl" flag will be executed. Command with "!snappmirror" and "!wafl" flag will be excluded.
	
	3. --diag-password
	With this option, user doesn't need to input the systemShell password during perfstat run.
	As all the nodes in the cluster will be having same systemShell password, provide only the systemShell password of a single node in case of C-Mode
	In case of 7-Mode, the node ip and its respective systemShell password should be provided. 
	Note: Patterns matching will not be case sensitive. 
	
	EXAMPLES FOR C-MODE
	
	%perfstat8 10.10.2.10 --mode="cluster-mode" 
	%perfstat8 10.10.2.10 --mode="cluster" 
	%perfstat8 10.10.2.10 --mode="c" 
	This command runs perfstat with 1 iterations. Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 
	
	%perfstat8 -i 5,10 --verbose 10.10.2.10 --mode="cluster-mode" 
	This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
	Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster.
	
	%perfstat8 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true" --mode="cluster-mode" 
	This command runs perfstat with default preset file and also collects the host side data along with the oracle AWR report. 
	Here 'someDbaUser' is a pre-existing DBA user. The another option for oracle params is '-w "oracle_login=testUser/testUser123" ' 
	
	%perfstat8 -i 5,10 --verbose 10.10.2.10 --enable-flag=snapmirror --mode="cluster-mode" 
	This command enables "snapmirror" flag in preset file and runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
	Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster.
	
	Example to provide systemShell password at command Line.
	%perfstat8 -i 5,10 --verbose 10.10.2.10 --diag-passwd=abcxyz --mode="cluster-mode" 
	This command runs perfstat with 5 iterations. Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 
	And "abcxyz" is the systemShell password.
	
	%perfstat8 --verbose --nodes=10.10.2.10,10.10.2.11,10.10.2.12 --diag-passwd=abcxyz --mode="cluster-mode" 
	This command runs perfstat with 1 iteration. Here IP address "10.10.2.10", "10.10.2.11", "10.10.2.12" are the address of a nodes in the cluster. 
	"abcxyz" is the systemShell password.
	
	EXAMPLES FOR 7-MODE
	
	%perfstat8 --nodes=10.72.57.222 --mode="7-mode" 
	%perfstat8 --nodes=10.72.57.222 
	This command runs perfstat with 1 iterations. Here "10.72.57.222" is the management IP for a 7-mode filer. 
	
	%perfstat8 -i 5,10 --verbose --nodes=10.72.57.222 --mode="7-mode" 
	This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration.  
	
	%perfstat8 -i 5,10 --verbose --nodes=10.72.57.222 --mode="7-mode" --enable-flag=snapmirror
	This command enables "snapmirror" flag in preset file and runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
	
	%perfstat8 --verbose --nodes=10.72.57.222 --mode="7-mode" -a "oracle" -w "runas=someDbaUser,sysdba=true" 
	This command runs perfstat with default preset file and also collects the host side data along with the oracle AWR report. 
	Here 'someDbaUser' is a pre-existing DBA user. The another option for oracle params is '-w "oracle_login=testUser/testUser123" ' 
	
	Example to provide systemShell password at command Line.
	%perfstat8 --verbose --nodes=10.10.2.10,10.10.2.11,10.10.2.12 --mode="7-mode" --diag-passwd=10.10.2.10:abcxyz1,10.10.2.11:abcxyz2,10.10.2.12:abcxyz3 
	This command runs perfstat with 1 iteration. Here IP address "10.10.2.10", "10.10.2.11", "10.10.2.12" are the address of the respective nodes. 
	"abcxyz1", "abcxyz2", "abcxyz3" are the systemShell passwords of the respective nodes.
	
	%perfstat8 --verbose --nodes=10.10.2.10 --mode="7-mode" --PROFILES=kahuna
	This command runs perfstat with 1 iteration. Here IP address "10.10.2.10" is the address of the node. 
	Example to show the usage of LEGACY MODE.
	%perfstat8 --verbose --nodes=10.10.2.10 --mode="7-mode" --LEGACY_MODE
	This command runs perfstat in Legacy Mode with 1 iteration. Here IP address "10.10.2.10", is the address of the node. 
	Note: System shell account for "diag" user needs to be setup before running perfstat with system shell commands.
	
	Note: Perfstat for Linux Platform only supports empty passphrases.

	Note: Converged perfstat does not accept empty password
	COPYRIGHT
	Copyright 2009, NetApp  All Rights Reserved. 


10.	EXAMPLES:
	---------
 

	
10.1.	Option : Cluster IP run

	Windows syntax : prompt% perfstat8.exe 10.10.2.10 													

	Linux syntax: prompt% perfstat8 10.10.2.10	   

	Description: 	This command runs perfstat with 1 iteration. Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster.


10.2.	Option:		7-mode default run
	
	Windows syntax :	prompt% perfstat8.exe --nodes=10.10.2.10 --mode="7-mode"	
	
	Linux syntax   :	prompt% perfstat8 --nodes=10.10.2.10 --mode="7-mode"	

	Description	   : 	This command runs perfstat with 1 iteration. Here IP address "10.10.2.10" is a management IP address for a 7-mode controller.	   
		

10.3.	Option:		--enable-flag	
	
	Windows syntax :	prompt% perfstat8.exe -i 5,10 10.10.2.10 --enable-flag=snapmirror	
	
	Linux syntax   :	prompt% perfstat8 -i 5,10 10.10.2.10 --enable-flag=snapmirror

	Description	   : This command enables "snapmirror" flag in preset file and runs perfstat with 5 iterations, waiting 10 seconds between each iteration.
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster.
		

10.4.	Option:	--preset-file	
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --preset-file user_defined_preset
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --preset-file user_defined_preset

	Description	   : 	This command collects data based on the preset file provided. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "user_defined_preset" is the user defined preset file.
	
			   
10.5.	Option:		-a and -w
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 -a "oracle" -w "runas=someDbaUser,sysdba=true"

	Description	   : 	This command collects oracle AWR report. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 'someDbaUser' is a pre-existing DBA user. 
				Another	option for oracle params is '-w "oracle_login=testUser/testUser123"	   

10.6.	Option:		-i (--iteration)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 .i 5,10
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 .i 5,10

	Description	   : 	This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration. 
				Here IP address "10.10.2.10" is a node-management
 				IP address of a node in the cluster.

		   

10.7.	Option:		--diag-passwd
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 .i 5,10 --diag-passwd=abcxyz
				prompt% perfstat8.exe .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=abcxyz
				prompt% perfstat8.exe .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=10.10.2.11:abcxyz 1, 10.10.2.12:abcxyz2
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 .i 5,10 --diag-passwd=abcxyz
				prompt% perfstat8 .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=abcxyz
				prompt% perfstat8 .-nodes=10.10.2.11, 10.10.2.12 .i 5,10 --diag-passwd=10.10.2.11:abcxyz 1, 10.10.2.12:abcxyz2

	Description	   : 	This command runs perfstat with 5 iterations, waiting 10 seconds between each iteration and with the system shell 
				password mentioned in the command line. 

				for c-mode:
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 
				10.10.2.11, 10.10.2.12 are the IP addresses of the nodes in the cluster. "abcxyz" is the systemshell password of the cluster.

				for 7-mode:
				10.10.2.11, 10.10.2.12 are the IP addresses of the nodes in the cluster. "abcxyz1", "abcxyz2" are the systemshell 
				passwords of the nodes	10.10.2.11 and 10.10.2.12.
	

10.8.	Option:		--PROFILES (-g)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --PROFILES=kahuna
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --PROFILES=kahuna .

	Description	   : 	This command runs perfstat with 1 iteration and collects the profile data. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "kahuna" is the profile domain.


10.9.	Option:		--MODULE_INST (-r)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --MODULE_INST=1
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --MODULE_INST=1 

	Description	   : 	This command runs perfstat with 1 iteration and collects the rastrace data. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and "1" is the module id.


10.10.	Option:		--SKTRACE_POINTS (-k)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --SKTRACE_POINTS="SK:7,WAFL:4"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --SKTRACE_POINTS="SK:7,WAFL:4" 

	Description	   : 	This command runs perfstat with 1 iteration and collects the SKTRACE data. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 


10.11.	Option:		--log
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --log
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --log

	Description	   : 	This command runs perfstat with 1 iteration and collects the log data. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 



10.12.	Option:		--hosts (-H)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --hosts=10.10.20.1:root:abcxyz, 10.10.20.2:root:abcpqr
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --hosts=10.10.20.1:root:abcxyz, 10.10.20.2:root:abcpqr

	Description	   : 	This command runs perfstat with 1 iteration and collects remote host data also. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 
				10.10.20.1, 10.10.20.2 are the IP addresses of the remote hosts.
				"root","abcxyz" and "abcpqr" are the username and passwords of the respective hosts.


10.13.	Option:		--switch (-S)
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --switch=10.10.20.1:BROCADE:root:abcxyz, 10.10.20.2:CISCO:root:abcpqr
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --switch=10.10.20.1:BROCADE:root:abcxyz, 10.10.20.2:CISCO:root:abcpqr

	Description	   : 	This command runs perfstat with 1 iteration and collects switch data. 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster and 
				10.10.20.1, 10.10.20.2 are the IP addresses of the switches. 
				"root","abcxyz" and "abcpqr" are the username and passwords of the respective switches. BROCADE and CISCO are the types of 
				switches.		   



10.14.	Option:		--exclude
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --exclude="TYPE=CONFIG"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --exclude="TYPE=CONFIG"

	Description	   :	This command runs perfstat with 1 iteration and exclude the commands whose TYPE is "CONFIG". 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster. 

	
10.15.	Option:		--include
	
	Windows syntax :	prompt% perfstat8.exe 10.10.2.10 --include="TYPE=CONFIG"
	
	Linux syntax   :	prompt% perfstat8 10.10.2.10 --include="TYPE=CONFIG"

	Description	   : 	This command runs perfstat with 1 iteration and includes only the commands whose TYPE is "CONFIG". 
				Here IP address "10.10.2.10" is a node-management IP address of a node in the cluster.



11.	FAQs:
	-----



11.1	Where to get plink.exe and puttygen.exe to generate SSH public/private key pair?

	Ans: 	These executables can be downloaded from this URL: 
         	 http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

11.2	 How to setup the ssh key on the controller?

	Ans: 	Here are the steps to do sshkey generation and exchange for windows. 
		For Linux hosts, "ssh-keygen" command can be used to generate the public/private key pair.

		Note: Ensure that you type a real value for the password and the password field is not left blank.
		Using SSH instead of RSH ensures security, but requires interactive authentication unless you first perform a host 
		key exchange. Using the free, open source utilities putty, puttygen, and plink, you can control the 
		controller from a Windows host and send non-interactive commands much like you would using RSH, but without the security risks.
		Enable SSH Host Key Exchange
		This document covers implementing SSH (using plink.exe, putty.exe, pscp.exe) as an alternative to RSH for security conscious environments.
	
	1.	Set up SecureAdmin on the controller (if you have not already done so).
	2.	Download putty.exe, plink.exe, puttygen.exe, pscp.exe and place in C:\windows or some directory in your system path.
	3.	Run puttygen (Click Start, Run, type puttygen, click OK).
	4.	When puttygen loads, click the Generate button. You will be prompted to move your mouse around the windows to 
		generate randomness. Make sure the Parameters are set as shown below (SSH-2 RSA).
	5.	At this point, we need to save our private key file somewhere on the host by clicking the Save Private Key button. 
		In this scenario, we chose C:\windows\privkey.ppk as the path/filename for my private key. Not the most secure location, but easy to remember.

		OPTIONAL - You may want to set an ACL on this file so that it can be accessed only by the users 
		that will connect via plink (or VFM using plink) as shown below:
		cacls c:\windows\privkey.ppk /g <YOURUSERNAME>:f <VFMUSERNAME>:f administrators:f 
		In the above example, plink.exe is being used both for the RSH replacement script (srsh.bat) as well as for VFM . 
		So, I want to grant full access to both of those users as well as the 'local administrators' group (and no-one else). 
		Of course, this step is optional, but you really should do something to	keep your private key file relatively secure.
	6.	Next, save your public key. Cut and paste the highlighted text into WordPad and save the file as authorized_keys.

		Note: DO NOT click save public key as this creates a format that the controller does not understand. 
		This file needs to be saved to: \\<CONTROLLERNAME>\c$\etc\sshd\<USERNAME>\.ssh\authorized_keys. 
		However, the problem is, there is no such folder. If you are using CIFS to copy the file, 
		this can be problematic as CIFS does not let you create folders with a "." at the beginning of their name.

		To get around this, create a folder named <USERNAME> in the following location using Windows Explorer: \\<CONTROLLERNAME>\c$\etc\sshd\
		e.g.- \\controllername\c$\etc\sshd\joeadmin
		Navigate to the newly created <USERNAME> folder using Windows Explorer and create a subfolder called ssh.

		FROM THE CONTROLLER CONSOLE, issue the following command:
		mv /etc/sshd/<USERNAME>/ssh /etc/sshd/[USERNAME]/.ssh
		Now you have the proper directory structure, so save the 'authorized_keys' file into this folder:
		\\<CONTROLLERNAME>\c$\etc\sshd\<USERNAME>\.ssh\authorized_keys
		We want to cache the controller's key in our registry, so issue the following command from the Windows system:
		plink -ssh -l root <CONTROLLERNAME> -i c:\windows\privkey.ppk
		You will be prompted to save the key to the local registry. Answer "yes" (this is required for VFM to work with plink.exe).
		SSH access is now enabled and commands can be sent to the controller by simply preceding them with ssh.


11.3	Generated a public SSH key with puttygen.exe and given it a passphrase but don.t know what to do with the saved key?

	Ans: Once the public/private key pair is generated by puttygen.exe, you should put the public key on the controller. 
	To achieve the same, please create the root directory under "/etc/sshd" directory and further create ".ssh" directory 
	under the newly created "root" directory. Once the "/etc/sshd/root/.ssh" directory is ready, please create the "authorized_keys" 
	file under this directory path by copying the content of the "public key" you just generated using puttygen.exe. 
	So after it, effectively you would have a file called "/etc/sshd/root/.ssh/authorized_keys" having the public key content.
	Now the private key from this public/private key pair should be used with the Perfstat command line.

11.4	Copied the plink.exe to the same folder as that of perfstat8, but don.t know what else to do with it?

	Ans:  Nothing is expected to be done explicitly after putting this exe. Perfstat would use it internally  to 
	execute the "parallel" commands, in case of 7-mode. 

11.5	The controller must have been added to the 'trusted known host list' on the client before running the perfstat. 
	Where exactly do we need to add to this trusted host list?

	Ans: The meaning for the phrase .trusted known host list. is that the server.s host "Keys" need to be cached in the system registry. 
	This requirement is imposed because Perfstat executes plink in 	batch mode. However, you cannot respond to the request to update Window's registry. 

	So precisely, to achieve the above, please run plink directly from a command prompt, and answer "y" when plink asks to update 
	the registry. Once the key is successfully cached, perfstat should be able to execute plink in batch mode 
	For example:
	%plink.exe -ssh -x -a -pw <your password for anyUser> -l <anyUser>  <your controller name/IP> "version"
	The server's host key is not cached in the registry. You  have no guarantee that the server is the computer you  think it is.
	The server's rsa2 key fingerprint is:
	ssh-rsa 768 67:4b:5a:0b:14:e9:5c:e0:65:b5:92:ef:48:26:92:38
 	If you trust this host, enter "y" to add the key to PuTTY's cache and carry on connecting.
	If you want to carry on connecting just once, without  adding the key to the cache, enter "n".
      	If you do not trust this host, press Return to abandon the connection.
      	Store key in cache? (y/n) y
      	NetApp Release R8.1RCxN_111122_0214 7-Mode: Tue Nov 22 02:32:29 PST 2011   <1>


11.6	How can we test the whether the public and private key pair transfer has been successful or not?

	Ans: Assuming you wanted to run perfstat without "password", so you need to modify the command line for the above run as follows:

	%perfstat8.exe -i 5,60 --nodes=<node-ip> --mode="7-mode" --sshprivatekey-file="priv_key.ppk"
	
	Please note that the above perfstat would run with "root" user. 
	If you want to run with some other user, you need to setup the key for that user and "-l" option need to be used at command line.


11.7	I am trying to run perfstat against a 7mode system. When it asks for the nodeshell user, 
	I give the systemshell diaguser name and password since there is no node shell in 7 mode that I know of?

	Ans: Nodeshell and Noderun shell are interchangeably used in 7-mode as default shell.  
	Being the systemshell user, "diag" is not exposed to Nodeshell user directly. So please run perfstat with any of 
	the user having administrative privilages ( could be find by the command "useradmin user list" on the controller).



11.8	What is this --SSH option for?

	Ans: In 7-Mode, the parallel commands will run either through "rsh" or "ssh".
	By default it is through "rsh". The prerequisite to run parallel commands using "rsh" is, 
	the local host details should be specified at "/etc/hosts.equiv" in the CONTROLLER.
	One can do this by using:
	wrfile -a /etc/hosts.equiv host_ip login_name
	prerequisite to use --SSH is:
	For windows, keep the plink.exe in the same location as that of the perfstat binary.

11.9	What exactly is this --time or -t option?

	Ans: --time option doesn't mean the time for which the perfstat will run or the time for which an iteration will run. 
	It specifies the sample time in minutes, per iteration. In other words, the time for which the parallel commands will run.

11.10	Is there a way that I can exclude or include specific commands for perfstat run?

	Ans: Yes. Using the --exclude or --include options one can exclude or include specific commands for perfstat run.
		For further information, please refer to the Converged Perfstat help manual.

11.11	Why is this plink.exe required?

	Ans: Converged perfstat tool opens a SSH connection and executes the PRESTATS and POSTSTATS commands over this channel. 
	Due to the limitation of 'only single interactive SSH session allowed' for 7-mode, the parallel commands 
	gets executed using "rsh.exe" on windows and "rsh binary on linux. There are few windows OS versions which does not have 
	native RSH support for example windows2008/windows7, to make parallel commands work on those windows clients an open 
	source command line tool called "plink.exe" for windows and "ssh" binary for linux is been used.


11.12	Why is the option "flat" used for profiling?

	Ans: profiling is triggered by means of --PROFILES or -g at command line. 
	These options take the profile domains as the input. "flat" is the mixture of the other domains like kahuna and storage. 

11.13	How can I collect sktrace data?
	Ans:  SKTRACE can be triggered by Converged Perfstat by using --SKTRACE_POINTS (or) -k. 
	The format should be <Module>:<debug level> (deafult: 'SK:7,WAFL:4'). The debug level should be in old data ONTAP7.x fashion

11.14	How can I make a customized preset file?

	Ans: The preset file is of the pattern:
	SECTION       TYPE    HARDWARE        OS      LICENSE SHELL   FLAG

	SHORTNAME   COMMAND

	For example:
	PRESTATS	PERF	NODE	*	*	NODERUN		

	options	options stats.wafltop.config volume,process,message.

	Each line in the preset file will be having these parameter tags. These tags are "tab" separated. "*" means all. " " means nothing.
	
	Note: profile and rastrace commands should be present in the customized preset file, if --PROFILES or --MODULE_INST options are used.

11.15  	What and why is this -g option used for?

	Ans: There are few windows OS versions which does not have native RSH support for example windows2008/windows 7. On these machines/others,
	using plink.exe for executing commands using .ssh. hangs. So, .openssh. is used to execute the commands using .g option. This .g option is 
	available only with --LEGACY_MODE and --sshprivatekey-file. Please refer to "openssh_procedure.txt" file to know the steps to follow, so as to make
	"-g" option work.

11.16  How to enable RSH on Mac OS X systems ?

	Ans: The below procedure is valid for Mac OS X 10.6.x onwards.

	1)  There are files, /System/Library/LaunchDaemons/login.plist and /System/Library/LaunchDaemons/shell.plist that control the rlogind and rshd daemons in 
	basically the same way as the files in /etc/xinetd.d. To enable those services, simply delete the lines,

	<key>Disabled</key>
	<true/>

	Note: In Mac OS x Lion, the *.plist files are provided as binary files, So they can not be edited with current form in any text editer. So they need to first be
	 converted into "XML" format {use plutil binary] and then delete those above lines.

       		1.a) $ sudo plutil -convert xml1 shell.plist
  	        1.b) $ sudo plutil -convert xml1 login.plist


	2) run $ sudo launchctl load /System/Library/LaunchDaemons/shell.plist
	3) run $ sudo launchctl load /System/Library/LaunchDaemons/login.plist
	4) run $ sudo launchctl list | egrep "rsh|login"
	5) you will see something like this when they are loaded:

	$ sudo launchctl list | egrep "rsh|login"
	-     0     com.apple.rlogind
	-     0     com.apple.rshd
	$


	6) run $ sudo launchctl start com.apple.rshd
	7) run $ sudo launchctl start com.apple.rlogind
	8) run $ sudo launchctl list | egrep "rsh|rlogin"
	--if the processes are properly started. 0 means they are loaded, >0 means they are started):

	$ sudo launchctl list | egrep "rsh|login"
	608     -     com.apple.rlogind
	604     -     com.apple.rshd
	$


	9)  run sudo su .
	10) run 'echo "localhost root" >> /etc/hosts.equiv'
	11) run rsh localhost :

		$ rsh localhost
		Last login: Tue Jan 8 23:21:04 on ttys000
		bash$

	12) Now revert back the xml format files [Only if done in step-1.a/1.b] into the original binary format.

		$ sudo plutil -convert binary1 shell.plist
		$ sudo plutil -convert binary1 login.plist
