NetApp Collector & Analyzer for Virtual Environments

A little bit over a year ago we undertook a project to develop a NetApp Collector & Analyzer tool that would be used to collect all storage relevant performance data from physical hosts in order to properly size our disk arrays for customers embarking on server virtualization projects. 

Now, one would ask why would NetApp try to re-invent the wheel given there are a lot of tools that can provide this capability such as VMware's Capacity Planner, Microsoft's Assessment and Planning Toolkit, and Windows native tools like Perfmon and Logman. The reason we decide to proceed with a project like this, is because either the above tools do not collect all the performance counters we need, and/or the analysis phase can be quite complex.

Having to go over 1000s of CSV files, analyze them, plot the output and decide on what values to input into our Sizing Tool is no simple task and takes time and resources. The reality is, the entire process can be quite a pain in the a$$.

So we developed a NetApp Collector and Analyzer Tool that will...collect tall he relevant performance data we're interested in and will perform analysis of 1000s of CSV files produced with the push of a button.

  This is an agentless collection with the requirement being windows user with WMI access. You can set things as the:

• Collection Interval
• Collection Duration
• Output Directory to dump a zip file containing the CSVs
• Can add individual hosts
• Can import Hosts from a CVS
• Can export hosts to a CSV
• Can encrypt host credentials read from a CSV

  After you've collected the data, you'd fire up the NetApp Analyzer tool. Prior to doing that, you want to unzip the zip file containing the CSVs. Then you'd start the Analyzer and you will point to to the directory the CSVs are located.

Depending on the Number of Hosts, Collection intervals and Duration, the amount of the CSVs and the size of each CSV will vary.  For example, for a 7 day run at 5 minute interval (default) for 49 hosts, I ended up with 1500 CSV files to be analyzed each of which had a 32k size. 

Under normal circumstances, analysis of 1500 files, would be a huge task, however using the NetApp Analyzer the entire process took about 15-20 minutes.

So what were the results I got back?

The above shows an overview of the results in terms of AVG and MAX numbers as well as the different workload characteristics. We would then take these number and input then into our NetApp Application Sizer which will produce a configuration interms of the Disk Array required and necessary capacity to sustain this load.

If you look at the above screenshot, there's a substantial deviation between the AVG and the MAX. So what would you size for? AVG or MAX? It depends.

From the above graph we see there are some spikes so we need to figure out what causes them by showing the data to the customer and asking questions. In this particular case, the spikes occur every day at approximately 9:00pm. These spikes are most likely due to some form of a backup procedure.

After, you decide, based on customer input, as to what to size for, you would use the above data points as input into the NetApp's Custom Application Sizer which will produce the necessary array and relevant configurations for the customer.

The NetApp Collector and Analyzer is available to all NetApp SEs and NetApp partners to assist their customers in server virtualization projects and proper storage sizing. It can be download from Field Portal. If you have difficulty locating it, search for "Collector". 

The NetApp Custom Application sizer is a web based tool also accessible to partners thru Field Portal.

Cheers

 

Technorati Tags: VMware,Hyper-V-Citrix,Performance Sizing,NetApp Collector & Analyzer,Server Virtualization Storage Sizing

Comments

[vikash roy]

Hello I am getting following error

[ Mon Sep 14 09:38:06 EDT 2009 ] ERROR [PlatformCommunicator$ConnectThread : r
un : ?] : 10.1.130.20 can not get result from Select Name from Win32_OperatingS
ystem
[ Mon Sep 14 09:38:06 EDT 2009 ] ERROR [PlatformCommunicator$ConnectThread : r
un : ?] : Exception thrown!
com4j.ComException: 800706ba The RPC server is unavailable. : The RPC server is
unavailable. : .\invoke.cpp:460
at com4j.Wrapper.invoke(Wrapper.java:122)
at $Proxy5.connectServer(Unknown Source)
at com.netapp.perf.vsizer.collector.RetrieveWMIbySelectWQL.retrieveWMIIn
foRowData(Unknown Source)
at com.netapp.perf.vsizer.collector.PlatformCommunicator$ConnectThread.r
un(Unknown Source)
Caused by: com4j.ComException: 800706ba The RPC server is unavailable. : The RPC
server is unavailable. : .\invoke.cpp:460
at com4j.Native.invoke(Native Method)
at com4j.StandardComMethod.invoke(StandardComMethod.java:95)
at com4j.Wrapper$InvocationThunk.call(Wrapper.java:258)
at com4j.Task.invoke(Task.java:44)
at com4j.ComThread.run0(ComThread.java:149)
at com4j.ComThread.run(ComThread.java:125)
[ Mon Sep 14 09:38:06 EDT 2009 ] ERROR [PlatformCommunicator$ConnectThread : r
un : ?] : 10.1.130.15 can not get result from Select Name from Win32_OperatingS
ystem
[ Mon Sep 14 09:38:06 EDT 2009 ] ERROR [PlatformCommunicator$ConnectThread : r
un : ?] : Exception thrown!
com4j.ComException: 800706ba The RPC server is unavailable. : The RPC server is
unavailable. : .\invoke.cpp:460
at com4j.Wrapper.invoke(Wrapper.java:122)
at $Proxy5.connectServer(Unknown Source)
at com.netapp.perf.vsizer.collector.RetrieveWMIbySelectWQL.retrieveWMIIn
foRowData(Unknown Source)
at com.netapp.perf.vsizer.collector.PlatformCommunicator$ConnectThread.r
un(Unknown Source)
Caused by: com4j.ComException: 800706ba The RPC server is unavailable. : The RPC
server is unavailable. : .\invoke.cpp:460
at com4j.Native.invoke(Native Method)
at com4j.StandardComMethod.invoke(StandardComMethod.java:95)
at com4j.Wrapper$InvocationThunk.call(Wrapper.java:258)
at com4j.Task.invoke(Task.java:44)
at com4j.ComThread.run0(ComThread.java:149)
at com4j.ComThread.run(ComThread.java:125)

I have added host and supplied root password

[Nick Triantos]

Hi,

It looks like either Windows Management Instrumentation (WMI) on the host may not be enabled or remote access to it is not enabled.

On the target server, go to Administrative Tools -> Computer Management.
2. Expand 'Services and Applications'
3. Right click for Properties on 'WMI Control'

Select the Security tab
5. Press the Security button

Add the monitoring user, and then be sure to check Remote Enable for the user/group that will be requesting WMI data.

At this point go back and see if this fixes the problem. It might take a couple of minutes.