HDS AMS 2000 Performance Analysis with PowerShell and PowerGadgets

Welcome!

It has been a while since my last post due to a very busy schedule with a SAN and virtualization project.

I have been working on an implementation of a HDS AMS 2500 midrange array for a VMWare vShere 4 environment. So far everything has been working and performing well. The management software included with the HDS AMS 2000 series array is SNM2(Storage Navigator Modular 2), A java based web application. This software also has a command line version which appears to be pretty comprehensive. It consists of a series of DOS executables, which can be run from PowerShell. There are a series of scripts I have been working on for viewing and creating storage resources on the array. I will share many of these in future posts. In this post I want to share some scripts I have written to extend the functionality of the performance monitoring utility in SNM2.

The base functionality of the array allows you to capture performance statistics to a text file. The file can be captured manually or automatically for a specified time period and interval down to one minute. One text file is produced per capture or all captures can be written to one file. Also, I believe based on the information I read in the SNM2 manual you can do some graphing with the web interface, but it requires an additional license and personally I think the PowerGadgets graphs are better.

The 4 scripts I have started with are get-performance_processor.ps1, get-performance_ports.ps1, get-performance_raidgroups.ps1, get-performance_luns.ps1, which do pretty much what they say and produce the following PowerGadgets charts.

Chart

The chart group is a tabbed interface which allows you to tab through the controllers and ports/RG/LU/Procs depending on the script being used. Each script generates different groups of charts for different performance counters. I have not implemented all of the performance counters just the ones which are most important to me now. I will be improving these scripts over time and implementing more counters. Here is an example of how the script works.

Script

After executing the script it will ask whether or not to collect data, if yes it will prompt for interval in minutes and time period. If no it will use previously collected data in the default output directory. Next it will ask to list data in text output. Then it will prompt for generation of each group of charts for ports, raid groups, luns or processors depending on the script run.

Now to the script. All of the scripts rely on the start-session.ps1 script and also require a password file be set for logging into the array. Additionally, an array has to be registered.

Example 1 shows a PowerShell script which will register an array and set the admin password.

You will need to replace ARRAYNAME, USERNAME and the IP Addresses for your environment.

 Example 2 shows the start-session PowerShell script which defines environmental information.

You will need to change the paths and ARRAYNAME for your environment.

Example 3 shows the get-performance_processor.ps1 script

This script collects the data from the array in separate files. Reads the pertinent data from the files and transforms it into object form which is fed into the PowerGadgets out-chart cmdlet. The other three scripts are longer as they digest more information.

To use these scripts you will need PowerShell, PowerGadgets( this a pay product with a free trial ), SNM2 CLI, and the script files attached to this post. Oh and an HDS AMS 2000 array.

Here are the script dowmloads
start-Session.txt
get-performance_processor.txt
get-performance_ports.txt
get-performance_raidgroups.txt
get-performance_luns.txt

Save the files to your script directory and change the extensions to .ps1

I hope someone finds this useful.

Regards,

Dave

Leave a Reply

Your email address will not be published. Required fields are marked *

3 comments

  1. Garret Black says:

    Hi David. We are using AMS2500’s in our environment and have already hit and resolved a few bottlenecks. The new one I uncovered is queue depth per lun. I was aware of queue depth of 512 per port. We have vSphere 4.1 with SIOC enabled but are still getting External I/O Detected alerts which is actually due to latency spikes even though all of the I/O in that disk pool comes from one Vmware cluster. It looks like i’m going to have to manually lower the queue depth of our esxi hosts due to our environment now growing to 16 hosts and 42 luns/datastores. I asked HDS if there was a way for me to monitor the tag count per lun in Tuning manager and he said there was not. Have you looked into monitoring tag count (queue depth) on the lun or port?

  2. dmuegge says:

    Garret,

    I have not tried to do that and I do not manage the AMS2500’s any longer. I did still have some old copies of pfmlogs and tag count values are accessable on a per lun basis.

    CTL LU Tag Count Average Tag Count
    0 0 1 0
    0 1 0 0
    0 2 0 0
    0 3 48 6
    0 4 0 0
    0 5 2 0
    0 6 0 0
    0 7 0 0
    0 8 0 0
    0 9 0 0
    0 10 0 0
    0 11 1 0
    0 12 0 0

    Sorry, I don’t have a script for you, but the one above could be used for the most part with some mods to the regex. The concept is the same.

    See the script above it is the auperform command in the HSNM2 CLI that will collect the info you need.

    Thanks,

    Dave

  3. Yves says:

    I test your script. ( port and processor) (AMS2100)
    y
    I have a problem with axe time.
    I haven’t time on axe X but I have the number of pfm0000.txt

    example:
    PS E:Program Files (x86)Storage Navigator Modular 2 CLI> cd test
    PS E:test> .portget-performance_ports.ps1
    Collect Data Y or N…
    Collection Time/Count in minutes: 2
    Collecting Data…
    Wed 08 28 16:25:44 2013 Start
    Wed 08 28 16:26:44 2013 Output File Name : pfm00000.txt Output Count : 1/2Turn…
    Wed 08 28 16:27:44 2013 Output File Name : pfm00001.txt Output Count : 2/2Turn…
    The performance statistics information file(s) have been outputted successfully.
    Analyzing Data…
    List Port Data Y or N…

    ctl port iops readiops writeiops readhit writehit tranrate readrate writerate datetim
    — —- —- ——– ——— ——- ——– ——– ——– ——— ——-
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0

    ctl port readcmdcount writecmdcoun readcmdhitco writecmdhitc readsize writesize datetime
    t unt ount
    — —- ———— ———— ———— ———— ——– ——— ——–
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0

    ctl port timeouterrcount datetime
    — —- ————— ——–
    0 A 0
    0 B 0
    1 A 0
    1 B 0
    0 A 0
    0 B 0
    1 A 0
    1 B 0
    0 A 0
    0 B 0
    1 A 0
    1 B 0
    0 A 0
    0 B 0
    1 A 0
    1 B 0
    0 A 0
    0 B 0
    1 A 0
    1 B 0

    Show Port IOPS Charts Y or N…

    I you send your email, I send to you my graph result.
    it’s no IO on my bay, no connect for the moment.

    file: pfm0000.txt
    No.1
    2013/08/28 16:25:45 – 2013/08/28 16:26:44 – SN:83050024
    —- Port Information —-
    CTL Port IO Rate(IOPS) Read Rate(IOPS) Write Rate(IOPS) Read Hit(%) Write Hit(%) Trans. Rate(MB/S) Read Trans. Rate(MB/S) Write Trans. Rate(MB/S)
    0 A 0 0 0 0 0 0 0 0
    0 B 0 0 0 0 0 0 0 0
    1 A 0 0 0 0 0 0 0 0
    1 B 0 0 0 0 0 0 0 0
    CTL Port Read CMD Count Write CMD Count Read CMD Hit Count Write CMD Hit Count Read Trans. Size(MB) Write Trans. Size(MB)
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    CTL Port CTL CMD IO Rate(IOPS) CTL CMD Trans. Rate(KB/S) CTL CMD Count CTL CMD Trans. Size(KB) CTL CMD Time(microsec.) CTL CMD Max Time(microsec.)
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    CTL Port Data CMD IO Rate(IOPS) Data CMD Trans. Rate(MB/S) Data CMD Count Data CMD Trans. Size(MB) Data CMD Time(microsec.) Data CMD Max Time(microsec.)
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    CTL Port Timeout Error Count
    0 A 0
    0 B 0
    1 A 0
    1 B 0
    CTL Port Random IO Rate(IOPS) Random Read Rate(IOPS) Random Write Rate(IOPS) Random Trans. Rate(MB/S) Random Read Trans. Rate(MB/S) Random Write Trans. Rate(MB/S)
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    CTL Port Random Read CMD Count Random Write CMD Count Random Read Trans. Size(MB) Random Write Trans. Size(MB)
    0 A 0 0 0 0
    0 B 0 0 0 0
    1 A 0 0 0 0
    1 B 0 0 0 0
    CTL Port Sequential IO Rate(IOPS) Sequential Read Rate(IOPS) Sequential Write Rate(IOPS) Sequential Trans. Rate(MB/S) Sequential Read Trans. Rate(MB/S) Sequential Write Trans. Rate(MB/S)
    0 A 0 0 0 0 0 0
    0 B 0 0 0 0 0 0
    1 A 0 0 0 0 0 0
    1 B 0 0 0 0 0 0
    CTL Port Sequential Read CMD Count Sequential Write CMD Count Sequential Read Trans. Size(MB) Sequential Write Trans. Size(MB)
    0 A 0 0 0 0
    0 B 0 0 0 0
    1 A 0 0 0 0
    1 B 0 0 0 0
    CTL Port XCOPY Rate(IOPS) XCOPY Read Rate(IOPS) XCOPY Write Rate(IOPS) XCOPY Read Trans. Rate(MB/S) XCOPY Write Trans. Rate(MB/S)
    0 A 0 0 0 0 0
    0 B 0 0 0 0 0
    1 A 0 0 0 0 0
    1 B 0 0 0 0 0
    CTL Port XCOPY Time(microsec.) XCOPY Max Time(microsec.)
    0 A 0 0
    0 B 0 0
    1 A 0 0
    1 B 0 0
    —- Cache Information —-
    CTL Write Pending Rate(%)
    0 0
    1 0
    CTL Clean Queue Usage Rate(%) Middle Queue Usage Rate(%) Physical Queue Usage Rate(%) Total Queue Usage Rate(%)
    0 0 0 0 0
    1 0 0 0 0
    CTL Partition Write Pending Rate(%)
    0 0 0
    0 1 0
    1 0 0
    1 1 0
    CTL Partition Clean Queue Usage Rate(%) Middle Queue Usage Rate(%) Physical Queue Usage Rate(%)
    0 0 0 0 0
    0 1 0 0 0
    1 0 0 0 0
    1 1 0 0 0
    —- Processor Information —-
    CTL Core Usage(%)
    0 X 0
    1 X 0
    CTL Host-Cache Bus Usage Rate(%) Drive-Cache Bus Usage Rate(%) Processor-Cache Bus Usage Rate(%)
    0 0 0 0
    1 0 0 0
    CTL Cache(DRR) Bus Usage Rate(%) Dual Bus Usage Rate(%) Total Bus Usage Rate(%)
    0 0 0 7
    1 0 0 7
    —- Drive Information —-
    CTL Unit HDU IO Rate(IOPS) Read Rate(IOPS) Write Rate(IOPS) Trans. Rate(MB/S) Read Trans. Rate(MB/S) Write Trans. Rate(MB/S) Online Verify Rate(IOPS)
    0 0 0 0 0 0 0 0 0 0
    0 0 1 0 0 0 0 0 0 0
    0 0 2 0 0 0 0 0 0 0
    0 0 3 0 0 0 0 0 0 0
    0 0 4 0 0 0 0 0 0 0
    0 0 5 0 0 0 0 0 0 0
    0 0 6 0 0 0 0 0 0 0
    0 0 7 0 0 0 0 0 0 0
    0 0 8 0 0 0 0 0 0 0
    0 0 9 0 0 0 0 0 0 0
    0 0 10 0 0 0 0 0 0 0
    0 0 11 0 0 0 0 0 0 0
    0 0 12 0 0 0 0 0 0 0
    0 0 13 0 0 0 0 0 0 0
    0 0 14 0 0 0 0 0 0 0
    1 0 0 0 0 0 0 0 0 0
    1 0 1 0 0 0 0 0 0 0
    1 0 2 0 0 0 0 0 0 0
    1 0 3 0 0 0 0 0 0 0
    1 0 4 0 0 0 0 0 0 0
    1 0 5 0 0 0 0 0 0 0
    1 0 6 0 0 0 0 0 0 0
    1 0 7 0 0 0 0 0 0 0
    1 0 8 0 0 0 0 0 0 0
    1 0 9 0 0 0 0 0 0 0
    1 0 10 0 0 0 0 0 0 0
    1 0 11 0 0 0 0 0 0 0
    1 0 12 0 0 0 0 0 0 0
    1 0 13 0 0 0 0 0 0 0
    1 0 14 0 0 0 0 0 0 0
    CTL Unit HDU Read CMD Count Write CMD Count Read Trans. Size Write Trans. Size Online Verify CMD Count
    0 0 0 0 0 0 0 42
    0 0 1 0 0 0 0 42
    0 0 2 0 0 0 0 42
    0 0 3 0 0 0 0 42
    0 0 4 0 0 0 0 42
    0 0 5 0 0 0 0 42
    0 0 6 0 0 0 0 42
    0 0 7 0 0 0 0 42
    0 0 8 0 0 0 0 42
    0 0 9 0 0 0 0 42
    0 0 10 0 0 0 0 42
    0 0 11 0 0 0 0 42
    0 0 12 0 0 0 0 42
    0 0 13 0 0 0 0 42
    0 0 14 0 0 0 0 42
    1 0 0 0 0 0 0 0
    1 0 1 0 0 0 0 0
    1 0 2 0 0 0 0 0
    1 0 3 0 0 0 0 0
    1 0 4 0 0 0 0 0
    1 0 5 0 0 0 0 0
    1 0 6 0 0 0 0 0
    1 0 7 0 0 0 0 0
    1 0 8 0 0 0 0 0
    1 0 9 0 0 0 0 0
    1 0 10 0 0 0 0 0
    1 0 11 0 0 0 0 0
    1 0 12 0 0 0 0 0
    1 0 13 0 0 0 0 0
    1 0 14 0 0 0 0 0
    —- Drive Operate Information —-
    CTL Unit HDU Operating Rate(%) Tag Count Unload Time(min.) Average Tag Count
    0 0 0 2 1 0 0
    0 0 1 2 1 0 0
    0 0 2 2 1 0 0
    0 0 3 2 1 0 0
    0 0 4 2 1 0 0
    0 0 5 2 1 0 0
    0 0 6 2 1 0 0
    0 0 7 2 1 0 0
    0 0 8 2 1 0 0
    0 0 9 2 1 0 0
    0 0 10 2 1 0 0
    0 0 11 2 1 0 0
    0 0 12 2 1 0 0
    0 0 13 2 1 0 0
    0 0 14 2 1 0 0
    1 0 0 0 1 0 0
    1 0 1 0 0 0 0
    1 0 2 0 0 0 0
    1 0 3 0 0 0 0
    1 0 4 0 0 0 0
    1 0 5 0 0 0 0
    1 0 6 0 1 0 0
    1 0 7 0 0 0 0
    1 0 8 0 0 0 0
    1 0 9 0 0 0 0
    1 0 10 0 0 0 0
    1 0 11 0 0 0 0
    1 0 12 0 1 0 0
    1 0 13 0 0 0 0
    1 0 14 0 0 0 0
    —- Backend Information —-
    CTL Path IO Rate(IOPS) Read Rate(IOPS) Write Rate(IOPS) Trans. Rate(MB/S) Read Trans. Rate(MB/S) Write Trans. Rate(MB/S) Online Verify Rate(IOPS)
    0 0 0 0 0 0 0 0 10
    0 1 0 0 0 0 0 0 0
    1 0 0 0 0 0 0 0 0
    1 1 0 0 0 0 0 0 0
    CTL Path Read CMD Count Write CMD Count Read Trans. Size Write Trans. Size Online Verify CMD Count
    0 0 0 0 0 0 630
    0 1 0 0 0 0 0
    1 0 0 0 0 0 0
    1 1 0 0 0 0 0