David Muegge's Blog http://muegge.com/blog IT Eclectia - A variety technical topics and tips. Fri, 04 Mar 2011 04:58:32 +0000 http://wordpress.org/?v=2.6.1 en VMWare SRM using HDS AMS 2000 http://muegge.com/blog/?p=142 http://muegge.com/blog/?p=142#comments Sun, 08 Aug 2010 17:41:44 +0000 dmuegge http://muegge.com/blog/?p=142 I have been working recently configuring VMWare SRM using Hitachi AMS 2000 arrays. This has not been the most straightforward or well documented process. I hope this post will save someone else a little time.

In this post I will focus on the configuration and the required prerequisites for the HDS SRA 2.0 for VMWare SRM. During testing I used ESX 4.0 update1, vCenter 4.0 update1 and SRM 4.01.

When working with this solution there are some obvious pieces of documentation such as the following:

Hitachi Storage Replication Adapter Software VMware vCenter Site Recovery Manager Deployment Guide

Site Recovery Manager Administration Guide

There are also some other helpful pieces of documentation which may not be as obvious, such as:

Hitachi AMS 2000 Family Command Control Interface (CCI) Installation, Reference, and User’s Guides. These can be found on the HDS Support portal.

Implementing VMware Site Recovery Manager with Hitachi Enterprise Storage Systems Whitepaper

How To Debug CCI Issues 1.3 article on the HDS GSC website

Also, the sample horcm.conf file installed with the CCI has some good information in the comments.

Now on to the configuration.

Prerequisites

Before configuration of the HDS SRA is possible the following requirements must be in place:

  • Two HDS AMS2000 arrays connected by WAN (FC or iSCSI)
  • One VMWare vCenter installation in the primary/protected site
  • One VMWare vCenter installation in the secondary/recovery site
  • One VMWare SRM installation in the primary/protected site
  • One VMWare SRM installation in the secondary/recovery site
  • TrueCopy replication in place for LUN’s with protected datastores
  • SRM Sites paired

Test Environment

Our test environment consists of one cluster at the primary site and one at the secondary site. We have a test SharePoint environment which is stored across four VMFS datastores. See the diagram below.

Test Environment

Our goal for this test configuration will be to failover the SharePoint environment to the recovery site. We are replicating the LUN’s containing all of the SharePoint system data using TrueCopy Extended Distance. Also SRM is installed at both sites and the sites have been paired.

After the items above are in place we can move on to the configuring the SRA.

HDS SRA 2.0 Configuration

When configuring the storage replication adapter the primary documentation is the deployment guide referenced above. I thought there were a few things missing from the document.

The first step in getting the SRA configured is making sure you have a copy of the proper HDS CCI for your array firmware. The HDS SRA relies on the Hitachi Command Control Interface, which must be installed on the SRM servers. I installed the CCI in the default c:\HORCM directory on both SRM servers. This is a straightforward install and is documented in the Hitachi AMS 2000 Family Command Control Interface (CCI) Installation Guide.

The portion of the CCI install that was tough for me was determining it needed to be installed as a service and creating the horcm.conf files. Our above example will only require two instances of the horcm service. One on each SRM server they will be HORCM0 and HORCM1.

To create the services we create the horcm_run.txt files and issue the following commands.

On the first SRM server - create the c:\HORCM\Tool\horcm0_run.txt file and the execute the following command:
    C:\HORCM\tool\svcexe /S=HORCM0 /A=C:\HORCM\Tool\svcexe.exe

On the second SRM server - create the c:\HORCM\Tool\horcm1_run.txt file and the execute the following command:
    C:\HORCM\tool\svcexe /S=HORCM1 /A=C:\HORCM\Tool\svcexe.exe

The horcmx_run.txt is created by making a copy of the file naming it appropriately and setting the HORCMINST variable to the correct instance number. This is documented in the file located in HORCM\Tool

After running these commands you should see the services appear in the windows services MMC

Then add the following lines to the %systemroot%\drivers\etc\services file on each SRM server.

1
2

horcm0     11000/udp        #horcm0 CCI service
horcm1     11001/udp        #horcm1 CCI service

The file should appear as below with one blank line below the last horcm service

1
2
3
4
5

rasadv                  9753/udp  
horcm0                 11000/udp         #horcm0 CCI service
horcm1                 11001/udp         #horcm1 CCI service
 
imip-channels          11320/tcp         #IMIP Channels Port

Once the services are installed the next step is to create the horcm.conf files. The first thing we need to do this is a command device. The SRA deployment guide left this step out. This is documented in the VMWare SRM with Enterprise Storage whitepaper mentioned earlier. Basically you create a small LUN and present it to the SRM server as a physical compatibility RDM. Then you initialize the disk and create a basic primary partition, but do not assign a drive letter or format it. I found one HDS document that said this LUN should be 33MB and one that said 36MB so I made it 40MB. Once this is done we have all that is needed to create the horcm.conf files.

HORCM0.conf on the SRM server at primary site

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

 
#/************************* For HORCM_MON *************************************/
HORCM_MON
#ip_address            service      poll(10ms)      timeout(10ms)
srm1.test.local        horcm0       8000            3000
 
 
#/************************** For HORCM_CMD ************************************/
HORCM_CMD
#dev_name          dev_name              dev_name
\\.\CMD-11111111-64
 
 
#/************************** For HORCM_LDEV ***********************************/
HORCM_LDEV
#dev_group            dev_name              Serial#       CU:LDEV(LDEV#)            MU#
DRLAB_SRM_TEST        DRLAB_AP_OS           11111111      0x003C
DRLAB_SRM_TEST        DRLAB_DB_OS           11111111      0x003D
DRLAB_SRM_TEST        DRLAB_OLTP_LOG        11111111      0x003E
DRLAB_SRM_TEST        DRLAB_OLTP_DATA       11111111      0x003F
 
 
#/************************* For HORCM_INST ************************************/
HORCM_INST
#dev_group           ip_address            service
DRLAB_SRM_TEST       srm2.test.local       horcm1 

HORCM1.conf on the SRM server at secondary site

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#/************************* For HORCM_MON *************************************/
HORCM_MON
#ip_address            service      poll(10ms)      timeout(10ms)
srm2.test.local        horcm1       8000            3000
 
 
#/************************** For HORCM_CMD ************************************/
HORCM_CMD
#dev_name          dev_name           dev_name
\\.\CMD-11111112-4
 
 
#/************************** For HORCM_LDEV ***********************************/
HORCM_LDEV
#dev_group                   dev_name             Serial#       CU:LDEV(LDEV#)       MU#
DRLAB_SRM_TEST               DRLAB_AP_OS          11111112      0x0001
DRLAB_SRM_TEST               DRLAB_DB_OS          11111112      0x0005
DRLAB_SRM_TEST               DRLAB_OLTP_LOG       11111112      0x0006
DRLAB_SRM_TEST               DRLAB_OLTP_DATA      11111112      0x0007
 
 
#/************************* For HORCM_INST ************************************/
HORCM_INST
#dev_group            ip_address        service
DRLAB_SRM_TEST        srm1.test.local   horcm0
 

A couple of points to note on these files is the relationship between the hosts and devices in the group. The HORCM_LDEV section on each instance contains a reference to the half of the pair it controls. The HORCM_INST section contains a reference to the opposite instance in each file. Also the command device naming format. It consists of “\\.\CMD-” followed by the array serial number and the LUN number “\\.\CMD-11111112-4″. Now that we have the configuration files we copy them into the %windir% on their respective SRM servers and start the services.

After this is complete we can install the SRA. This is downloaded from the VMWare website and the executable is named RMHTCSRA.exe. It is a simple, no option, install. After this there are some environment variable which need to be set.

setx SplitReplication true /m
setx RMSRATMU 1 /m

Then reboot the SRM servers. We are now ready to configure the SRA using the SRM plug-in in vCenter.

Here we see the paired sites in site recovery.

 SRM Sites

Click on configure array managers and we see the following dialog.

 SRA Config 1

Click Add to add a protected site array manager and we see the following configuration dialog.

 SRM Config 2

We enter the name and HORCMINST=0 for the first instance on the primary server in the protected site. The we use a different name and HORCMINST=1 for the next instance on the secondary server in the recovery site. Here we see both sides configured.

 SRA Final ConfigSRA Final Config 2

The last step in the wizard allows us to confirm the SRA sees the replicated datastores properly.

 Replicated Datastores

We see the LUN numbers match the devices in the horcm<x>.conf files. The datastore group in this diagram consists of four LUN’s which also belong to the same TCE consistency group. These are the LUN’s being used by our test SharePoint application and database servers.

At this point we are now ready to complete configuration of the protection groups and recovery plans in Site Recovery Manager. The process for configuring these is documented in the SRM administration guide. Protection groups are configured at the protected site and recovery plans are configured at the recovery site. Here is a screenshot of the test recovery plan for our SharePoint environment.

Recovery Plan 1 

When we run a test on this recovery plan we can see the test runs successfully and waits for us to complete testing before clicking continue to return to a ready state.

Recovery Plan 2 

During this phase we can look at a couple of things to confirm what is happening in the process. One is the new datastores we will see in the configuration tab of the DR ESX hosts.

Datastore Snapshots/Replicas 

There was no need to change the LVM.EnableResignature or LVM.DisallowSnapshotLun settings at the host level in ESX 4 as this is enabled at the volume level and SRM handles this at the time of testing or failover. Another part of the process we can confirm at this time is the status of the TrueCopy pairs. Here we see the pairs are in split status.

TrueCopy Split Status

Now we can complete any other testing to confirm success of the test and then click continue in the recovery plan to return to a ready state. After the test completes we can see the datastores are removed from the recovery ESX hosts and the TCE pairs are returned to a paired status after resynchronization.

After the testing process is completed we can review some of the steps in the SRM logs. The logs are located under %allusersprofile%\ VMware\VMware vCenter Site Recovery Manager\Logs. These log entries and the HORCM logs under c:\HORCM\log are the primary sources of information in troubleshooting problems with this process.

I hope someone finds this post useful. Next I am going to be testing this with secondary copies at the recovery site using Shadowimage and Copy-on-Write.
 

Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=142 HDS AMS 2000 Storage Resource Reporting with PowerShell http://muegge.com/blog/?p=128 http://muegge.com/blog/?p=128#comments Sun, 29 Nov 2009 22:16:57 +0000 dmuegge http://muegge.com/blog/?p=128 Welcome!

I have been creating a few PowerShell scripts for use with the HDS AMS 2000 array. One thing I found I needed was a quick way to look at DP(Dynamic Provisioning) pools, raid groups, and LUNs. I also wanted to be able to see the associations and filter easily. Since I am working on a new deployment I have been creating Raid Groups and Luns often. I needed a quick way to see what I currently had while creating new resources.

I created a PowerShell script that would quickly show existing resources by raid group or DP pool. It also uses nickname info for the devices that are maintained in three csv files(LU_Nicknames.csv,RG_Nicknames.csv,DP_Nicknames.csv). These are simple comma delimited text files which contain the ID and nickname of each resource. The files are updated as storage resources are added. This allows me to easily identify the resources and to filter for specific devices.

The script executes three HSNM2 CLI commands and reads the information into object form. The LUN information is then shown grouped by raid group or DP pool.

Here is the output with the nickname search parameter set to “DB”. This will return all database resources based on the naming standard. If this is left null it will return all resources.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

PS D:\DTools\Scripts\PS\HDS-SNM2> D:\DTools\Scripts\PS\HDS-SNM2\get-LunsByDPRG.ps1
Nickname Search: DB
 
-- DP Pool  Number: 0  Nickname: DB_SQL_OLTP  Raid Level: 5( 5D+1P)  Type: SAS  Capacity: 1986.0 GB  Used: 734.0 GB ----------------------------------------------------------------
 
 
lu                                      nickname                                capacity                                stripesize                              status
--                                      --------                                --------                                ----------                              ------
3                                       DB01_OLTP_DATA                          300.0                                   256                                     Normal
20                                      DB01_OLTP_DATA                          300.0                                   256                                     Normal
22                                      DB01_OLTP_DATA                          300.0                                   256                                     Normal
29                                      DB01_DSS_DATA_TEMP                      600.0                                   256                                     Normal
 
 
 
-- RAID GROUP  Number: 4  Nickname: DB_LOG  Raid Level: 1+0( 2D+2D)  Type: SAS  Capacity: 535.7 GB  Free: 0.0 GB -------------------------------------------------------------------
 
 
lu                                      nickname                                capacity                                stripesize                              status
--                                      --------                                --------                                ----------                              ------
5                                       DB01_OLTP_LOG                           267.0                                   64                                      Normal
21                                      DB01_OLTP_LOG                           268.7                                   64                                      Normal
 
 
 
-- RAID GROUP  Number: 7  Nickname: DB_OS  Raid Level: 5( 3D+1P)  Type: SAS  Capacity: 398.4 GB  Free: 183.4 GB --------------------------------------------------------------------
 
 
lu                                      nickname                                capacity                                stripesize                              status
--                                      --------                                --------                                ----------                              ------
9                                       DB01_OS_DATA                            215.0                                   256                                     Normal
 
 
 
-- RAID GROUP  Number: 8  Nickname: DB_LOG  Raid Level: 1+0( 2D+2D)  Type: SAS  Capacity: 535.7 GB  Free: 535.7 GB -----------------------------------------------------------------

Here is the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

# -- Setup login info and SNM2 environment ------------------
. ./start-session.ps1
Set-Location $env:STONAVM_HOME
 
# -- Import Nickname tables ---------------------------------------
$LUNicknames = Import-Csv $SCRIPTHOME\LU_Nicknames.csv
$RGNicknames = Import-Csv $SCRIPTHOME\RG_Nicknames.csv
$DPNicknames = Import-Csv $SCRIPTHOME\DP_Nicknames.csv
 
# -- Get storage resource info -----------------------------------
$luresults = & ./auluref.exe -unit $DEFAULTARRAY -g -nosublu -totalsize
$rgresults = & ./aurgref.exe -unit $DEFAULTARRAY -g
$dpresults = & ./audppool.exe -unit $DEFAULTARRAY -refer -g
 
# -- Setup object collections -----------------------------------
$allrgdata = @()
$allludata = @()
$alldpdata = @()
 
# -- Get LUN data -----------------------------------------------------------
foreach($luresult in $luresults){
 
	if ($luresult -match 'LU\s+Capacity\s+Size\s+Group\s+Pool\s+Level\s+Type\s+Status'){
 
		$LUDataRead = $TRUE
 
	}elseif($LUDataRead){
 
		if($luresult -match '(\d+)\s+(\d+\.\d+)\sGB\s+(\d+)KB\s+(\d+|N/A)\s+(\d+|N/A)\s+(([5,6]|1\+0)\(\s+\d+D\+\d+[D,P]\))\s+(SAS|SATA)\s+(.+)'){
			$ludata = "" | Select lu,nickname,capacity,stripesize,raidgroup,dppool,raidlevel,type,status
			$ludata.lu = $matches[1]
			$Nickname = $LUNicknames | where {$_.LU -eq $ludata.lu} | select nickname
			$ludata.nickname = $Nickname.nickname
			$ludata.capacity = $matches[2]
			$ludata.stripesize = $matches[3]
			$ludata.raidgroup = $matches[4]
			$ludata.dppool = $matches[5]
			$ludata.raidlevel = $matches[6]
			$ludata.type = $matches[8]
			$ludata.status = $matches[9]
			$allludata += $ludata
		}
	}
}
 
# --  Get Raid Group data --------------------------------------------------------
foreach($rgresult in $rgresults){
 
	if ($rgresult -match 'Group\s+Level\s+Groups\s+Type\s+Total\sCapacity\s+Free\sCapacity\s+Priority\s+Status'){
 
		$RGDataRead = $TRUE
 
	}elseif($RGDataRead){
 
		if($rgresult -match '(\d+)\s+(([5,6]|1\+0)\(\s+\d+D\+\d+[D,P]\))\s+(\d+)\s+(SAS|SATA)\s+(\d+\.\d+)\sGB\s+(\d+\.\d+)\sGB\s\(\s*\d+\.{1}\d+%{1}\)\s+(.+\s.+)\s+(.+)'){
 
			$rgdata = "" | Select rg,nickname,raidlevel,raidconfig,paritygroups,type,totalcapacity,freecapacity,priority,status
			$rgdata.rg = $matches[1]
			$Nickname = $RGNicknames | where {$_.RG -eq $rgdata.rg} | select nickname
			$rgdata.nickname = $Nickname.nickname
			$rgdata.raidlevel = $matches[2]
			$rgdata.raidconfig = $matches[3]
			$rgdata.paritygroups = $matches[4]
			$rgdata.type = $matches[5]
			$rgdata.totalcapacity = $matches[6]
			$rgdata.freecapacity = $matches[7]
			$rgdata.priority = $matches[8]
			$rgdata.status = $matches[9]
			$allrgdata += $rgdata
		}	
	}
}
 
 
# --  Get DP pool data -------------------------------------------------------------
foreach($dpresult in $dpresults){
 
	if ($dpresult -match 'Pool\s+Level\s+Total\sCapacity\s+Consumed\sCapacity\s+Type\s+Status'){
 
		$DPDataRead = $TRUE
 
	}elseif($DPDataRead){
 
		if($dpresult -match '(\d+)\s+(([5,6]|1\+0)\(\s+\d+D\+\d+[D,P]\))\s+(\d+\.\d+)\sGB\s+(\d+\.\d+)\sGB\s+(SAS|SATA)\s+(.+)'){ 
 
			$dpdata = "" | Select dp,nickname,raidlevel,totalcapacity,capacityused,type,status
			$dpdata.dp = $matches[1]
			$Nickname = $DPNicknames | where {$_.DP -eq $dpdata.dp} | select nickname
			$dpdata.nickname = $Nickname.nickname
			$dpdata.raidlevel = $matches[2]
			$dpdata.totalcapacity = $matches[4]
			$dpdata.capacityused = $matches[5]
			$dpdata.type = $matches[6]
			$dpdata.status = $matches[7]
			$alldpdata += $dpdata
		}	
	}
}
 
 
# -- Get search string -----------------------------
$searchstring = Read-Host "Nickname Search"
 
# -- Display LUN's by DP pool --------------------------------------------------------------
foreach($dppool in $alldpdata){
 
	$dpnum = $dppool.dp
	$dpcapacity = $dppool.totalcapacity
	$dpused = $dppool.capacityused
	$dpnickname = $dppool.nickname
	$dplevel = $dppool.raidlevel
	$dptype = $dppool.type
 
	if($dpnickname -like "*$searchstring*"){
		$message = "-- DP Pool  Number: $dpnum  Nickname: $dpnickname  Raid Level: $dplevel  Type: $dptype  Capacity: $dpcapacity GB  Used: $dpused GB -"
		$charadd = (180 - $message.length)
		for($x=1;$x -le $charadd;$x++){$message += "-"}
		Write-Host ""
		Write-Host $message
		Write-Host ""
	}
	if($searchstring){
		$allludata | where {(($_.nickname -like "*$searchstring*") -and ($_.dppool -eq $dpnum))} | FT -property lu,nickname,capacity,stripesize,status
	}
	else
	{
		$allludata | where {$_.dppool -eq $dpnum} | FT -property lu,nickname,capacity,stripesize,status
	}	
}
 
# -- Display LUN"s by raid group -------------------------------------------------
foreach($rgroup in $allrgdata){
 
	$rgnum = $rgroup.rg
	$rgcapacity = $rgroup.totalcapacity
	$rgfree = $rgroup.freecapacity
	$rgnickname = $rgroup.nickname
	$rglevel = $rgroup.raidlevel
	$rgtype = $rgroup.type
 
	if($rgnickname -like "*$searchstring*"){
		$message = "-- RAID GROUP  Number: $rgnum  Nickname: $rgnickname  Raid Level: $rglevel  Type: $rgtype  Capacity: $rgcapacity GB  Free: $rgfree GB -"
		$charadd = (180 - $message.length)
		for($x=1;$x -le $charadd;$x++){$message += "-"}
		Write-Host ""
		Write-Host $message
		Write-Host ""
	}
	if($searchstring){
		$allludata | where {(($_.nickname -like "*$searchstring*") -and ($_.raidgroup -eq $rgnum))} | FT -property lu,nickname,capacity,stripesize,status
	}
	else
	{
		$allludata | where {$_.raidgroup -eq $rgnum} | FT -property lu,nickname,capacity,stripesize,status
	}
}
 
# -- CD back to script directory and set window title ---------------------
Set-Location $SCRIPTHOME
$Host.UI.RawUI.WindowTitle = "DP-RG-LU Info"

The script uses the start-session.ps1 file to establish connectivity with the HDS array. Additional information regarding the use of this include file can be found at this post. Then the script executes HSNM2 CLI commands to return information on DP pools raid groups and LUNS. The script uses regular expressions to parse the output and convert it into objects. It also reads in the nickname files and adds the data to the custom objects.

The objects are then output using the built-in PowerShell formating engine with a little custom formating thrown in for the group headers.

Here is an example nickname file:

1
2
3
4
5
6
7

RG,Nickname
0,MG_OS
1,AP_OS
2,DT_OS
3,DB_SQL_OLTP
7,DB_OS
8,DB_LOG

I suppose this may not be necessary with the use of Device Manager, but I am still learning it and I could not quite get this view with it. Besides I am more of a scripting kind of guy. I also really like the output of this script as it gives me just the view of the array I need when I am allocating new storage and setting up new resources. I use this script in conjuction with two other scripts for creating LUN’s and raid groups. I plan to post those scripts soon.

Hope this helps,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=128 HDS AMS 2000 Performance Analysis with PowerShell and PowerGadgets http://muegge.com/blog/?p=114 http://muegge.com/blog/?p=114#comments Sat, 24 Oct 2009 01:53:49 +0000 dmuegge http://muegge.com/blog/?p=114 Welcome!

It has been a while since my last post due to a very busy schedule with a SAN and virtualization project.

I have been working on an implementation of a HDS AMS 2500 midrange array for a VMWare vShere 4 environment. So far everything has been working and performing well. The management software included with the HDS AMS 2000 series array is SNM2(Storage Navigator Modular 2), A java based web application. This software also has a command line version which appears to be pretty comprehensive. It consists of a series of DOS executables, which can be run from PowerShell. There are a series of scripts I have been working on for viewing and creating storage resources on the array. I will share many of these in future posts. In this post I want to share some scripts I have written to extend the functionality of the performance monitoring utility in SNM2.

The base functionality of the array allows you to capture performance statistics to a text file. The file can be captured manually or automatically for a specified time period and interval down to one minute. One text file is produced per capture or all captures can be written to one file. Also, I believe based on the information I read in the SNM2 manual you can do some graphing with the web interface, but it requires an additional license and personally I think the PowerGadgets graphs are better.

The 4 scripts I have started with are get-performance_processor.ps1, get-performance_ports.ps1, get-performance_raidgroups.ps1, get-performance_luns.ps1, which do pretty much what they say and produce the following PowerGadgets charts.

Chart

The chart group is a tabbed interface which allows you to tab through the controllers and ports/RG/LU/Procs depending on the script being used. Each script generates different groups of charts for different performance counters. I have not implemented all of the performance counters just the ones which are most important to me now. I will be improving these scripts over time and implementing more counters. Here is an example of how the script works.

Script

After executing the script it will ask whether or not to collect data, if yes it will prompt for interval in minutes and time period. If no it will use previously collected data in the default output directory. Next it will ask to list data in text output. Then it will prompt for generation of each group of charts for ports, raid groups, luns or processors depending on the script run.

Now to the script. All of the scripts rely on the start-session.ps1 script and also require a password file be set for logging into the array. Additionally, an array has to be registered.

Example 1 shows a PowerShell script which will register an array and set the admin password.

1
2
3
4
5
6
7

$env:STONAVM_HOME="C:\Program Files\Storage Navigator Modular 2 CLI"
$env:STONAVM_ACT="on"
$env:STONAVM_RSP_PASS="on"
$env:LANG="en"
cd "C:\Program Files\Storage Navigator Modular 2 CLI"
./auunitadd -unit ARRAYNAME -ctl0 192.168.1.1 -ctl1 192.168.1.2
./auaccountenv -set -uid USERNAME

You will need to replace ARRAYNAME, USERNAME and the IP Addresses for your environment.

 Example 2 shows the start-session PowerShell script which defines environmental information.

1
2
3
4
5
6
7
8
9

# Comand Enviroment
$env:STONAVM_HOME="C:\Program Files\Storage Navigator Modular 2 CLI"
$env:STONAVM_ACT="on"
$env:STONAVM_RSP_PASS="on"
$env:LANG="en"
 
# Global
$SCRIPTHOME = "C:\Scripts\HDS-SNM2"
$DEFAULTARRAY = "ARRAYNAME"

You will need to change the paths and ARRAYNAME for your environment.

Example 3 shows the get-performance_processor.ps1 script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

 
# -- User Defined Constants ---------------------------
$LOG_PATH = "C:\Scripts\HDS-SNM2\pfmlog"
 
# -- Modify controller and core options as needed -----
$Controllers = @(0,1)
$cores = @("X","Y")
 
# -- Modifications below this line should not be required ---------------------------------------
 
 
 
 
# -- Setup environment ------------------------------
$testsnapin = $null
$testsnapin = get-pssnapin | where { $_.Name -eq "PowerGadgets"}
if(-not $testsnapin){add-pssnapin -Name PowerGadgets}
. ./start-session.ps1
Set-Location $env:STONAVM_HOME
 
 
# --  Object to hold processor usage data ------------------------
$allprocusagedata = @()
 
# -- Processing flags -------------------------------------
$PROCUsageDataRead = $FALSE
 
 
# -- Specify whether to run new collection if not existing files are used --
Write-Host ”Collect Data Y or N...”
$modekey = $Host.UI.RawUI.ReadKey(”NoEcho,IncludeKeyDown”)
if($modekey.Character.ToString().ToLower() -eq "y"){
	$collectiontime = Read-Host "Collection Time/Count in minutes"
	Write-Host "Collecting Data..."
	./auperform.exe -unit $DEFAULTARRAY -auto 1 -count $collectiontime -path $LOG_PATH -pfmstatis	
}
 
 
# -- Get data from files --------------------------------------------
Write-Host "Analyzing Data..."
$perffiles = Get-ChildItem $LOG_PATH
 
foreach($perffile in $perffiles){
	$filedata = Get-Content $LOG_PATH\$perffile
	foreach($line in $filedata){
 
		# Get collection times
		if($line -match '^(\d{4}\/(?:0[1-9]|1[0-2]?)\/(?:0[1-9]|[12][0-9]|3[01]?)\s(?:0[0-9]|1[0-9]|2[0-3]?)\:(?:0[0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]?)\:(?:0[0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]?))\s-\s(\d{4}\/(?:0[1-9]|1[0-2]?)\/(?:0[1-9]|[12][0-9]|3[01]?)\s(?:0[0-9]|1[0-9]|2[0-3]?)\:(?:0[0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]?)\:(?:0[0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]?))$'){
 
			$collectfrom = $matches[1]
			$collectto = $matches[2]				
		}
 
		# Get Proc Usage Data
		if ($line -match 'CTL\s+Core\s+Usage'){
 
			$PROCUsageDataRead = $TRUE
 
		}elseif($PROCUsageDataRead){
 
			$line -match '([0,1]{1})\s+([X,Y]{1})\s+(\d+)' | Out-Null 
			$procusagedata = "" | Select ctl,core,usage,datetime
			$procusagedata.ctl = $matches[1]
			$procusagedata.core = $matches[2]
			$procusagedata.usage = $matches[3]
			$procusagedata.datetime = $collectto
			$allprocusagedata += $procusagedata
 
			if(($procusagedata.ctl -eq 1) -and ($procusagedata.core -eq "Y")){$PROCUsageDataRead = $FALSE}
		}
	}
}
 
 
# -- List Data ---------------------------------
Write-Host ”List Processor Data Y or N...”
$modekey = $Host.UI.RawUI.ReadKey(”NoEcho,IncludeKeyDown”)
if($modekey.Character.ToString().ToLower() -eq "y"){
	$allprocusagedata | ft -prop *
}
 
 
# -- Create Processor Usage Chart Group ----------------------
Write-Host ”Show Processor Usage Charts Y or N...”
$modekey = $Host.UI.RawUI.ReadKey(”NoEcho,IncludeKeyDown”)
if($modekey.Character.ToString().ToLower() -eq "y"){
	Write-Host "Generating Charts..."
	foreach($Controller in $Controllers){
		foreach($core in $cores){
			$ChartTitle = "Processor Usage Controller:$Controller Core:$core"
			$allprocusagedata | where {($_.core -eq $core)-and ($_.ctl -eq $Controller)} | sort datetime | Select core,usage,datetime | `
			Out-Chart -Values usage -Series_0_Text "Processor Usage %" -Label datetime `
			-Title $ChartTitle -Gallery Lines -Group "PROCUSAGE" -Name $ChartTitle -Caption "Processor Usage %"
		}
	}
}
 
Set-Location $SCRIPTHOME

This script collects the data from the array in separate files. Reads the pertinent data from the files and transforms it into object form which is fed into the PowerGadgets out-chart cmdlet. The other three scripts are longer as they digest more information.

To use these scripts you will need PowerShell, PowerGadgets( this a pay product with a free trial ), SNM2 CLI, and the script files attached to this post. Oh and an HDS AMS 2000 array.

Here are the script dowmloads
start-Session.txt
get-performance_processor.txt
get-performance_ports.txt
get-performance_raidgroups.txt
get-performance_luns.txt

Save the files to your script directory and change the extensions to .ps1

I hope someone finds this useful.

Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=114 Exchange DB Reporting with PowerShell and Log Parser http://muegge.com/blog/?p=112 http://muegge.com/blog/?p=112#comments Thu, 02 Jul 2009 01:09:51 +0000 dmuegge http://muegge.com/blog/?p=112 I ran across a useful post today as I was roaming through Google Analytics.

Using PowerShell, LogParser and PowerGadgets to get Exchange 2003 storage information – Part 1

Wes Stahler uses Log Parser and PowerShell to report on the free space in an Exchange Database.

This is a task I have done in the past. I will add this script to my toolkit.

Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=112 ADAM Administration with SharePoint and PowerShell http://muegge.com/blog/?p=101 http://muegge.com/blog/?p=101#comments Mon, 11 May 2009 01:39:14 +0000 dmuegge http://muegge.com/blog/?p=101 Welcome!

Recently I worked on finding a simple way to create a web based administrative interface for an ADAM directory. The requirements were to create a simple web based interface to allow business personnel to manage users and groups for an application directory. It was also desirable if this solution would easily integrate with SharePoint.

After doing a little searching on the web, I found a combination that fit the bill.

The Quest AD Management Shell CmdLets – This is a PowerShell Snap-In that allows administration of AD and ADAM. The cmdlets are from Quest Software you can find more info here. I have used them in other scripts and they have come in very handy. To make these work in this solution from SharePoint, the Quest Snapin .dll and it’s dependants need to be copied to the global assembly cache and entered as a safecontrol in the sharepoint web.config.

The iLoveSharePoint PowerWebPart 3.0 – This is a web part which allows the execution of PowerShell code from the web part. This web part is from the CodePlex project iLoveSharePoint by Christian Glessner. I was impressed with this web part. It is easy to install and configure and relatively simple to use.

The PowerWebPart allows you to execute scripts that will render asp.net web controls in the web part. This allows you to retrieve user input from the controls to use as script inputs. The possibilities are endless. For my purposes I only needed a very simple user interface.

I wanted a way to use this for different ADAM partitions so I tried to allow for different configuration scripts. The design I decided on consisted of three levels of scripts one for configuration one for data access and one UI script for each web part.

The code sample below is the configuration and connection script. This script defines the user and group containers and the directory connect and disconnect functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# ---------------------------------------------------------------------------
### <Library name='ADAM_Config_Lib.ps1'>
### <Author>David Muegge</Author>
### <Description>
### 	Adam connection and configuration library
### </Description>
### <Dependencies>
###	Quest AD CmdLets 1.2
### </Dependencies>
### </Library>
# ---------------------------------------------------------------------------
$SnapInName = "Quest.ActiveRoles.ADManagement"
$testsnapin = $null
$testsnapin = get-pssnapin | where { $_.Name -eq $SnapInName}
if(-not $testsnapin){add-pssnapin -Name $SnapInName}
 
# Container paths
$usercontainer = "cn=Users,cn=ADAMDEV"
$groupcontainer = "cn=Groups,cn=ADAMDEV"
 
 
function connect-ADAM{
 
	# This allows the use of a windows account for ADAM athentication without granting the application pool account rights to ADAM
	$passwd = Get-Content "C:\testpwd.txt" | convertto-securestring
	$adamcred = new-object -typename System.Management.Automation.PSCredential -argumentlist "TestSVR1\adamadmin",$passwd
 
	connect-QADService -Service localhost -Credential $adamcred
 
 
}
 
function disconnect-ADAM{
 
	param($adamconnection)
 
	Disconnect-QADService -Connection $adamconnection
 
}

The next script is the function library for data access to the ADAM directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

# ---------------------------------------------------------------------------
### <Library name='ADAM_Lib.ps1'>
### <Author>David Muegge</Author>
### <Description>
### 	Adam directoy function library
### </Description>
### <Dependencies>
###	Quest AD CmdLets 1.2
### </Dependencies>
### </Library>
# ---------------------------------------------------------------------------
function get-Users{
 
	$adamconnection = connect-ADAM
	$results = get-QADObject -SearchRoot $usercontainer -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
	return $results
 
 
}
 
 
function get-Groups{
 
	$adamconnection = connect-ADAM
	$results = get-qadgroup -SearchRoot $groupcontainer -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
	return $results
 
 
}
 
 
function new-User{
 
	param([String]$logonid)
 
	# Random password generator here
	$newpassword = "password123$"
 
	$adamconnection = connect-ADAM
	New-QADUser -Name $newlogonid -UserPassword $newpassword -ParentContainer $usercontainer -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
 
}
 
function delete-user{
 
	param([String]$logonid)
 
	$adamconnection = connect-ADAM
	$UserDN = "CN=$logonID," + $usercontainer
	Remove-QADObject -Identity $UserDN -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
 
}
 
 
function new-Group{
 
	param([String]$groupid)
 
	$adamconnection = connect-ADAM
	New-QADGroup -Name $groupid -GroupType "Default" -ParentContainer $groupcontainer -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
}
 
 
function delete-group{
 
	param([String]$groupid)
 
	$adamconnection = connect-ADAM
	$GroupDN = "CN=$groupid," + $groupcontainer
	Remove-QADObject -Identity $GroupDN -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
 
}
 
 
function Add-UserToGroup{
 
	param([string]$username,[String]$groupname)
 
	$adamconnection = connect-ADAM
	# Get user dn
	$results = Get-QADUser -Name $username -Connection $adamconnection
	foreach($result in $results){$userdn = $result.dn.ToString()}
 
	# Get Group DN
	$results = Get-QADGroup -Name $groupname -Connection $adamconnection
	foreach($result in $results){$groupdn = $result.dn.ToString()}
 
	# Add User to Group
	Add-QADGroupMember -Identity $groupdn -Member $userdn -Connection $adamconnection
	disconnect-ADAM -adamconnection $adamconnection
 
}

The next script is an example of a UI script for the web part. When a new PowerWebPart is created a template script is added by default. This script provides a framework and some sample code. Christian also has an add-on which allows you to use PowerGui to edit your script from SharePoint. The entire solution contains one script similar to this for each web part.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

########### Initialize ############
. c:\ADAMAdmin\ADAM_Lib.ps1
 
### declare global variables and functions
$groupnametext = New-Object System.Web.UI.WebControls.TextBox
$createbutton = New-Object System.Web.UI.WebControls.Button
$groupnamelabel = New-Object System.Web.UI.WebControls.Label
$deletebutton = New-Object System.Web.UI.WebControls.Button
 
 
############## Load ##############
 
### first time the OnLoad fires before CreateChildControls
#function OnLoad
#{
#	# Check if GET Request (first request). 
#	if($isPostBack -eq $false)
#	{
#		$label.Text = 'GET request.'
#	}
#}
 
######## 3. Create Controls ########
 
## create child controls
function CreateChildControls($controls)
{
	$controls.Add($groupnametext)	
 
	$createbutton.Text = 'Add Group'
	Subscribe-Event $createbutton 'Click' 'OnCreateButtonClicked'
  	$controls.Add($createbutton)
 
	$deletebutton.Text = 'Delete Group'
	Subscribe-Event $deletebutton 'Click' 'OnDeleteButtonClicked'
  	$controls.Add($deletebutton)
 
	$groupnamelabel.Text = "Groupname"
	$controls.Add($groupnamelabel)
}
 
######## Events ########
 
## handle control events
## subscribe to an event with "Subscribe-Event($control, 'eventName','callback function name')"
function OnCreateButtonClicked($sender, $Args)
{
 
   $newgroupid = $groupnametext.text
   # Create new group
   new-Group -groupid $newgroupid
   $groupnametext.text = [string]::Empty
 
 
}
 
function OnDeleteButtonClicked($sender, $Args)
{
 
   $newgroupid = $groupnametext.text
   # Delete group
   Remove-Group -groupid $newgroupid
   $groupnametext.text = [string]::Empty
 
 
}
 
########## Render  #########
 
### render html
function Render($writer)
{
    $writer.Write("<table><tr><td>")
	$groupnamelabel.RenderControl($writer)
	$writer.Write("</td><td>")		
	$groupnametext.RenderControl($writer)
	$writer.Write("</td></tr><tr><td>&nbsp</td><td colspan=2>")
  	$createbutton.RenderControl($writer)
	$writer.Write("&nbsp")
	$deletebutton.RenderControl($writer)
  	$writer.Write("</td><td></td></tr></table>")
 
}
 
######### Error Handling #########
 
### render custom error message
##function OnError($exception, $writer){}
 
########## Unload #########
 
### clean up
##function OnUnload()
##{
##}

The screenshot below shows the complete solution. This method was simple, effective and easy to create. I dot sourced the corresponding web part script and the connection script in each web part.

This is a pretty quick and easy way to expose some simple administrative or user functionality on a SharePoint Intranet.

I hope this helps.

Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=101 Connection History with PowerShell and NetStat http://muegge.com/blog/?p=99 http://muegge.com/blog/?p=99#comments Fri, 20 Mar 2009 04:55:34 +0000 dmuegge http://muegge.com/blog/?p=99 Welcome!

This is a little trick some might find useful. I was working on decommissioning some servers and I needed a way to find out what was connecting to these machines. I decided to create a script to log connections. I have done this in the past in various ways which usually involved logging a bunch of data and then querying against it to find the unique connections.

This time it finally occurred to me, just filter the data as it is being collected. So I set out to write a PowerShell script that would keep a running list of client TCP connections to a given machine. This information would be stored in a text file.

The first step was to collect the information and put it into a PowerShell object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Create array to hold netstat data
$allnetstatdata = @()
 
# Get TCP connections from netstat
$netstatResults = & netstat -p TCP 
foreach($line in $netstatResults){
 
	# Create object data and add to object
	$netstatdata = "" | Select local,port,foreign
	if(-not (($line.Trim() -eq [String]::Empty) -or ($line.contains("Active Connections")) -or ($line.contains("Local Address")))){
		if ($line -match '\s*(TCP)\s{1,}(\D\S{1,})\s{1,}(\D\S{1,})\s{1,}(\D\S{1,}$)'){
			$localhost = $matches[2].Split(":")
			$netstatdata.local = $dchost[0]
			$netstatdata.port = $dchost[1]
			$foreignhost = $matches[3].Split(":")
			$netstatdata.foreign = $foreignhost[0]
			$allnetstatdata = $allnetstatdata + $netstatdata
		}
	}			
}

Then the next step was to read the file with the previous information and add it to the PowerShell object. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Define Log File
$logfile = C:\TCPConnections.txt
 
# Get data from file and add to object
$filedata = Get-Content $logfile
if($filedata){
	foreach($line in $filedata){
		$netstatdata = "" | Select local,port,foreign
		$entry = $line.Split("`t")
		$netstatdata.port = $entry[0]
		$netstatdata.local = $entry[1]
		$netstatdata.foreign = $entry[2]
		$allnetstatdata = $allnetstatdata + $netstatdata
	}
}

We can now remove the duplicates from the combined information and save the updated file. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Get unique rows
$newnetstatdata = $allnetstatdata | Select-Object * -unique 
 
# Add combined data to string
$tempcontent = New-Object System.Text.StringBuilder
foreach($netstat in $newnetstatdata){
	if($netstat.port){$tempcontent.Append($netstat.port.ToString()) | Out-Null}
	$tempcontent.Append("`t") | Out-Null 
	if($netstat.local){$tempcontent.Append($netstat.local.ToString()) | Out-Null}
	$tempcontent.Append("`t") | Out-Null 
	if($netstat.foreign){$tempcontent.Append($netstat.foreign.ToString()) | Out-Null}
	$tempcontent.Append("`n") | Out-Null 
}
 
# Write to file
Set-Content -path $logfile $tempcontent.ToString()

We can run this script in a scheduled task at whatever interval is required. Now we have a log of unique inbound TCP connections.

Best Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=99 PowerShell, Log Parser, PowerGadgets, and GeoIP what fun! http://muegge.com/blog/?p=93 http://muegge.com/blog/?p=93#comments Sun, 15 Mar 2009 01:03:53 +0000 dmuegge http://muegge.com/blog/?p=93 Welcome!

One day I was pondering how I might use log parser to map visitors to a website by state. I am aware this is easily done with tools like Google Analytics, but I was interested in using existing logs for the info.

Using the PowerShell and Log Parser functions from the library listed in a previouse post. Log parser can easily get the visitors by IP Address from an IIS log.

# Get unique ip addresses to working file
$query = @"
SELECT Distinct c-ip As IPAddress
INTO D:\Unique_IP_Addresses.txt
From C:\Logs\ex080310.log
"@
$LPInputFormat = Get-LPInputFormat "w3c"
$LPOutputFormat = Get-LPOutputFormat "csv"
Invoke-LPExecuteBatch -query $query -inputtype $LPInputFormat -outputtype $LPOutputFormat | Out-Null

The next task is to get the location of the IP addresses. The tool I chose for this task was the free GeoLite City from MaxMind http://www.maxmind.com/app/geolitecity. Here is an example:

# Get location for IP addresses from GEOIP DB
$geodbpath = "C:\GeoLiteCity"
$visitors = Import-Csv -Path "D:\Unique_IP_Addresses.txt"
$ipinfo = New-Object -ComObject "GeoIPCOMEx.GeoIPEx"
$ipinfo.set_db_path($geodbpath) | Out-Null
Add-Content -Path "D:\IP_Locations.txt" "Location"
foreach($visitor in $visitors){
	$ipinfo.find_by_addr($visitor.ipaddress) | Out-Null
	if($ipinfo.Region){
		Add-Content -Path "D:\IP_Locations.txt" $ipinfo.Region
	}
	else
	{
		Add-Content -Path "D:\IP_Locations.txt" "Unknown"
	}
}
 
# Get State counts
$query = @"
SELECT Location, Count(Location) As Visitors
INTO D:\VisitorsByState.txt
From D:\IP_Locations.txt
Group By Location
"@
$LPInputFormat = Get-LPInputFormat "csv"
$LPOutputFormat = Get-LPOutputFormat "csv"
Invoke-LPExecuteBatch -query $query -inputtype $LPInputFormat -outputtype $LPOutputFormat | Out-Null

There are a couple of ways to use the MaxMind GeoIP database. It can be used in it’s native binary format or it can be imported into SQL from csv files. MaxMind recommends using the binary format, which is what I chose to do. MaxMind also provides API’s for use with a variety of platform’s. I chose to use the COM version.

After the location is determined the counts are calculated. This brings us to the point where we need to chart the results. The tool I chose for this operation is PowerGadgets. This is a charting tool made for use with PowerShell, it can be handy. Here is an example:

# Create PowerGadgets Map
$charttitle = "Daily visitors by state"
$visits = Import-Csv -Path D:\VisitorsByState.txt
$visits | Out-Map -mapsource "US\USA-StatesAbrev" `
		-label location `
		-values visitors `
		-palette "highcontrast" `
		-output $chartout `
		-size 600,400 `
		-BackColor White `
		-Titles_0_BackColor White `
		-Titles_0_Font "Arial, 14" `
		-Titles_0_Text $charttitle `
		-Titles_0_TextColor Black `
		-conditionalattributes_0_condition_from 1 `
		-conditionalattributes_0_condition_to 10 `
		-conditionalattributes_0_color Red `
		-conditionalattributes_0_text "1 to 10" `
		-conditionalattributes_1_condition_from 11 `
		-conditionalattributes_1_condition_to 25 `
		-conditionalattributes_1_color Blue `
		-conditionalattributes_1_text "11 to 25" `
		-conditionalattributes_2_condition_from 26 `
		-conditionalattributes_2_condition_to 50 `
		-conditionalattributes_2_color Green `
		-conditionalattributes_2_text "26 to 50" `
		-conditionalattributes_3_condition_from 51 `
		-conditionalattributes_3_condition_to 100 `
		-conditionalattributes_3_color Yellow `
		-conditionalattributes_3_text "51 to 100" `
		-conditionalattributes_4_condition_from 101 `
		-conditionalattributes_4_condition_to 1000 `
		-conditionalattributes_4_color Violet `
		-conditionalattributes_4_text "101 to 1000"

 

And here is our final Chart.

 

This works pretty well the only drawback to this solution is that PowerGadgets is a pay tool, but since I own a copy it suites my needs.

Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=93 Automated Machine Builds with PowerShell and AutoIT http://muegge.com/blog/?p=73 http://muegge.com/blog/?p=73#comments Sun, 22 Feb 2009 01:41:07 +0000 dmuegge http://muegge.com/blog/?p=73 I have been using PowerShell recently to create scripted builds for virtual machines. This method provides some benefits over imaging.

  • A single base image per OS can be used
  • Applications which have issues with imaging can be easily installed such as SQL Server, BizTalk, Exchange, Etc.
  • Scripted installs can be changed or updated easier than images
  • Additional configuration changes such as registry settings and/or configuration files can be easily done


One challenge I faced with this process was automating the installation of applications which do not have unattended install capabilities.

A great tool I found for this was AutoIT. It provides a method to script the windows GUI and enables automating those installs. AutoIT consists of a COM object, scripting language, editor, compiler, and a cool window info tool. I will not get into the details of AutoIT here it has good documentation. I found PowerShell to be much stronger at the scripting part, but the COM object provides excellent window control functionality to PowerShell. The window info tool that comes with AutoIT is also very handy. Here is a simple example:

1
2
3
4
5
6

	$AutoIt = New-Object -ComObject AutoItX3.Control
	$result = $AutoIt.Run("notepad.exe")
 
	$wintitle = "Untitled - Notepad"
	$AutoIt.WinWaitActive($wintitle) | Out-Null
	$AutoIt.Send("This is a test")

To run the code above you will need to install AutoIT or just register the AutoItX.dll COM object. The code will launch notepad and type “This is a test”. While this is a pretty useless example it shows the basic purpose of AutoIT and how it is used from PowerShell.

The next piece I looked at was creating a function library to make it easy to use AutoIT from PowerShell.

Example: AutoIT PowerShell function library

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346

# Installation path
$InstallDir = '\\SERVERNAME\Installs$'
 
# -- Copy and register AutoIt COM object -------------------------------------
Copy-item $InstallDir\AutoItX3.dll $env:windir\system32
& regsvr32 AutoItX3.dll /s
 
# Wait for registration and create AutoIT instance
Start-Sleep 2
$AutoIt = New-Object -ComObject AutoItX3.Control
 
# ---------------------------------------------------------------------------
### <Function name='Test-ProcessRunning'>
### <Author>David Muegge</Author>
### <CreateDate>20080510</CreateDate>
### <Description>
### 	Test if process in running give name
### </Description>
### <Usage>
### 	$result = Test-ProcessRunning -processname <string>
### </Usage>
### <Notes>
### 	Helper function used to wait for process to complete
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Test-ProcessRunning([String]$processname){
 
	$currentprocess = Get-Process | where {$_.Name -eq $processname}
 
	if($currentprocess){
		return 0
	}else{
		return 1
	}	
}
 
# ---------------------------------------------------------------------------
### <Function name='Test-PIDRunning'>
### <Author>David Muegge</Author>
### <CreateDate>20080510</CreateDate>
### <Description>
### 	Test if process in running given process ID
### </Description>
### <Usage>
### 	$result = Test-PIDRunning
### </Usage>
### <Notes>
### 	Helper function used to wait for process to complete
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Test-PIDRunning([Int]$pidnumber){
 
	$currentprocess = Get-Process | where {$_.ID -eq $pidnumber}
 
	if($currentprocess){
		return 0
	}else{
		return 1
	}	
}
 
 
# ---------------------------------------------------------------------------
### <Function name='Start-WindowControl'>
### <Author>David Muegge</Author>
### <CreateDate>20080516</CreateDate>
### <Description>
### 	Wait for window presence and activate
### </Description>
### <Usage>
### 	Start-WindowControl -wintitle <window title> -wintext <window text> -keyarray <("v1","v2","v3")>
### </Usage>
### <Notes>
### 	The sleeps and the winactivate winwaitactive sequence was required for greater reliability.
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Start-WindowControl{
 
	param([String]$wintitle,
	   	  [String]$wintext=$null,
		  [String[]]$keyarray=$null)
 
	# Window text was supplied
	if($wintext){
		do{Start-Sleep -Milliseconds 100}until($AutoIt.WinExists($wintitle,$wintext))
		$AutoIt.WinActivate($wintitle,$wintext) | Out-Null
		Start-Sleep -Milliseconds 200
		$AutoIt.WinWaitActive($wintitle,$wintext) | Out-Null
		Start-Sleep -Milliseconds 200
	}
	else
	{
		# Only window title was supplied
		do{Start-Sleep -Milliseconds 100}until($AutoIt.WinExists($wintitle))
		$AutoIt.WinActivate($wintitle) | Out-Null
		Start-Sleep -Milliseconds 200
		$AutoIt.WinWaitActive($wintitle) | Out-Null
		Start-Sleep -Milliseconds 200	
	}
 
 
	# If keys were supplied
	if($keyarray){
 
		# Send Keys
		foreach($key in $keyarray){
			$AutoIt.Send($key)
		}
	}
 
}
 
# ---------------------------------------------------------------------------
### <Function name='Start-WindowControlAB'>
### <Author>David Muegge</Author>
### <CreateDate>20081125</CreateDate>
### <Description>
### 	Wait for window presence and activate
###		will accept one of two windows texts
### </Description>
### <Usage>
### 	Start-WindowControlAB -wintitle <window title> -wintext <window text> -altwintext <alternate window text> -keyarray <("v1","v2","v3")>
### </Usage>
### <Notes>
### 	The sleeps and the winactivate winwaitactive sequence was required for greater reliability.
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Start-WindowControlAB{
 
	param([String]$wintitle,
	   	  [String]$wintext=$null,
		  [String]$altwintext=$null,
		  [String[]]$keyarray=$null)
 
		# Wait for window to exist and activate
		$winfoundflag = $false
		do{
			Start-Sleep -Milliseconds 200
			if($AutoIt.WinExists($wintitle,$wintext)){$winfoundflag = $true}
			if($AutoIt.WinExists($wintitle,$altwintext)){$winfoundflag = $true;$wintext = $altwintext}
 
		}until($winfoundflag)
		$AutoIt.WinActivate($wintitle,$wintext) | Out-Null
		Start-Sleep -Milliseconds 200
		$AutoIt.WinWaitActive($wintitle,$wintext) | Out-Null
		Start-Sleep -Milliseconds 500
 
		# If keys were supplied
		if($keyarray){
		# Send Keys
			foreach($key in $keyarray){
				$AutoIt.Send($key)
			}
		}
}
 
# ---------------------------------------------------------------------------
### <Function name='Start-WindowControlQAB'>
### <Author>David Muegge</Author>
### <CreateDate>20081125</CreateDate>
### <Description>
### 	Wait for window presence and activate
###		will accept one of two windows texts and Title texts
### </Description>
### <Usage>
### 	Start-WindowControlQAB -wintitle <window title> -altwintitle <window title> -wintext <window text> -altwintext <alternate window text> -keyarray <("v1","v2","v3")>
### </Usage>
### <Notes>
### 	The sleeps and the winactivate winwaitactive sequence was required for greater reliability.
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Start-WindowControlQAB{
 
	param([String]$wintitle,
		  [String]$altwintitle,
	   	  [String]$wintext=$null,
		  [String]$altwintext=$null,
		  [String[]]$keyarray=$null)
 
	# Wait for window to exist and activate
	$winfoundflag = $false
	do{
		Start-Sleep -Milliseconds 200
		if($AutoIt.WinExists($wintitle,$wintext)){$winfoundflag = $true}
		if($AutoIt.WinExists($altwintitle,$altwintext)){$winfoundflag = $true;$wintext = $altwintext;$wintitle = $altwintitle}
 
	}until($winfoundflag)
	$AutoIt.WinActivate($wintitle,$wintext) | Out-Null
	Start-Sleep -Milliseconds 200
	$AutoIt.WinWaitActive($wintitle,$wintext) | Out-Null
	Start-Sleep -Milliseconds 500
 
	# If keys were supplied
	if($keyarray){
	# Send Keys
		foreach($key in $keyarray){
			$AutoIt.Send($key)
		}
	}
}
 
 
 
# ---------------------------------------------------------------------------
### <Function name='Start-WindowControlN'>
### <Author>David Muegge</Author>
### <CreateDate>20080516</CreateDate>
### <Description>
### 	Checks for window presence once a second for specified number of times
###		Sends keys when found
### </Description>
### <Usage>
### 	Start-WindowControlN -check <integer> -wintitle <window title> -wintext <window text> -keyarray <@("'n")>
### </Usage>
### <Notes>
### 	The sleeps and the winactivate winwaitactive sequence was required for greater reliability.
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Start-WindowControlN{
 
	param([Int32]$check,
		  [String]$wintitle,
	   	  [String]$wintext=$null,
		  [String[]]$keyarray=$null)
 
	# Check if window exists once a second for N times
	for($i = 0; $i -lt $check; $i++){Start-Sleep -Seconds 1
		if($AutoIt.WinExists($wintitle,$wintext)){
			$AutoIt.WinActivate($wintitle,$wintext)
			Start-Sleep -Milliseconds 100
			$AutoIt.WinWaitActive($wintitle,$wintext) | Out-Null 
			Start-Sleep -Milliseconds 200
 
			# If keys were supplied
			if($keyarray){
			# Send Keys
				foreach($key in $keyarray){
					$AutoIt.Send($key)
				}
			}
 
			break
		}
	}
}
 
# ---------------------------------------------------------------------------
### <Function name='Start-WindowControlS'>
### <Author>David Muegge</Author>
### <CreateDate>20080516</CreateDate>
### <Description>
### 	Wait for window presence and activate
### </Description>
### <Usage>
### 	Start-WindowControlS -wintitle <window title> -wintext <window text> -keyarray <("v1","v2","v3")>
### </Usage>
### <Notes>
### 	The sleeps and the winactivate winwaitactive sequence was required for greater reliability.
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Start-WindowControlS{
 
	param([String]$wintitle,
		  [String]$wintext=$null,
		  [String[]]$keyarray=$null)
 
	# Window text was supplied	 
	if($wintext){
		do{
			Start-Sleep -Milliseconds 200
		}until($AutoIt.WinExists($wintitle,$wintext))
 
		do{
			$AutoIt.WinActivate($wintitle,$wintext) | Out-Null
			Start-Sleep -Milliseconds 200
			$AutoIt.WinWaitActive($wintitle,$wintext) | Out-Null
			Start-Sleep -Milliseconds 500
 
			# If keys were supplied
			if($keyarray){
 
				# Send Keys
				foreach($key in $keyarray){
					$AutoIt.Send($key)
				}
			}
 
		}until(-not $AutoIt.WinExists($wintitle,$wintext))
 
	}
	else
	{
 
		# Only window title was supplied
		do{
			Start-Sleep -Milliseconds 200
		}until($AutoIt.WinExists($wintitle))
 
		do{
			$AutoIt.WinActivate($wintitle) | Out-Null
			Start-Sleep -Milliseconds 200
			$AutoIt.WinWaitActive($wintitle) | Out-Null
			Start-Sleep -Milliseconds 500
 
			# If keys were supplied
			if($keyarray){
 
				# Send Keys
				foreach($key in $keyarray){
					$AutoIt.Send($key)
				}
			}
		}until(-not $AutoIt.WinExists($wintitle))
	}
}
 
# ---------------------------------------------------------------------------
### <Function name='Write-Log'>
### <Author>David Muegge</Author>
### <CreateDate>20080517</CreateDate>
### <Description>
### 	Writes output to host with timestamp
### </Description>
### <Usage>
### 	Write-Log <logentry>
### </Usage>
### <Notes>
### 	Intended for us with Transcript functionality
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Write-Log{
 
	param([String]$logentry)
 
	$logtime = Get-Date -Format T
	Write-Host "$logtime - $logentry"
 
}

Then using the functions from the library above I created application installation functions for various packages.

Example: PowerShell Application Installation Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# ---------------------------------------------------------------------------
### <Function name='Install-TortoiseSVN152'>
### <Author>David Muegge</Author>
### <CreateDate>20080826</CreateDate>
### <Description>
### 	Installs Tortoise Subversion Client 1.5.2.13595-win32-svn-1.5.1
### </Description>
### <Usage>
### 	$result = Install-TortoiseSVN152
### </Usage>
### <Notes>
### 	Requires reboot when complete
###		Will not always prompt for reboot and will restart automatically
###		Always put at end of install step
### </Notes>
### </Function>
# ---------------------------------------------------------------------------
function Install-TortoiseSVN152{
 
	param([switch]$restart)
 
	$result = $AutoIt.Run("msiexec /i $InstallDir\TortoiseSVN-1.5.2.13595-win32-svn-1.5.1.msi")
	Write-Log "Tortoise Subversion Client 1.5.2 Install Launched PID:$result"
 
	Start-WindowControl -wintitle "TortoiseSVN 1.5.2.13595 (32 bit) Setup" -wintext "Welcome to the TortoiseSVN 1.5.2.13595 (32 bit) Setup Wizard" -keyarray @("!n")
	Start-WindowControl -wintitle "TortoiseSVN 1.5.2.13595 (32 bit) License Agreement" -wintext "I &accept the terms in the License Agreement" -keyarray @("!a","!n")
	Start-WindowControl -wintitle "TortoiseSVN 1.5.2.13595 (32 bit) Setup" -wintext "Custom Setup" -keyarray @("!n")
	Start-WindowControl -wintitle "TortoiseSVN 1.5.2.13595 (32 bit) Setup" -wintext "Ready to Install" -keyarray @("!i")
	Start-WindowControl -wintitle "TortoiseSVN 1.5.2.13595 (32 bit) Setup" -wintext "Click the Finish button to exit the Setup Wizard." -keyarray @("!f")
	Start-WindowControl -wintitle "Changelog.txt - Notepad" -keyarray @("!{F4}")
	Start-WindowControl -wintitle "Installer Information" -keyarray @("&Yes")
 
	# Check if reboot window exists once a second for five seconds
	Start-WindowControlN -check 5-wintitle "Installer Information" -wintext "&Yes" -keyarray @("!n")
 
	Write-Log "Tortoise Subversion Client 1.5.2 Install Completed"
 
	# Restart if selected
	if($restart){
		Write-Host "$logtime - Restarting..."
		Shutdown /r /t 1 /d P:2:4
	}
}

Using the method above I created a library of PowerShell installation functions to be called from automated build scripts.

This seems to be working pretty well so far and it allows automated builds to be scripted quickly. PowerShell and AutoIT work well together, the AutoIT COM object provides excellent GUI control and PowerShell provides strong scripting and debugging features.

Best Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=73 Log Parser and PowerShell – Part II http://muegge.com/blog/?p=65 http://muegge.com/blog/?p=65#comments Sat, 22 Nov 2008 02:05:34 +0000 dmuegge http://muegge.com/blog/?p=65 Welcome back!

Last post I talked about using the Log Parser executable from PowerShell. I also briefly mentioned the Log Parser COM component. In this post I will go into more depth on using the COM component from PowerShell.

The COM component exposes a simple object model consisting of only three main objects.

  • LogQuery – Object used to execute queries and batches
  • LogRecordSet – Object returned by LogQuery.Execute method
  • LogRecord – Child object of LogRecordset. Object returned by LogRecordSet.getRecord method

There are also objects containing the input and output formats. There are several of them and they are well documented in the Log Parser documentation. I will give examples as we progress.

The first step on my task to integrate with the Log Parser COM object was to put together a function library of the basic building blocks.

The library consists of the following functions.

  • Get-LPInputFormat – Returns a Log Parser input format object
  • Get-LPOutputFormat – Returns a Log Parser output format object
  • Invoke-LPExecute – Executes the LogQuery.Execute method and returns a LogRecordSet
  • Invoke-LPExecuteBatch – Executes the LogQuery.ExecuteBatch method to output the query in the requested format
  • Get-LPRecordSet – Executes a Log Parser Query and returns a LogRecordSet as an array of PowerShell objects
  • Get-LPRecord – Returns LogRecord object as a PowerShell object from the current record of a LogRecordSet object

With these functions we can support almost all of the Log Parser functionality in PowerShell. I did not build in support for the custom COM input type or the NAT and Datagrid output types. The NAT and Datagrid output types can be handled in a different way in PowerShell. The COM input format is a challenge I left for another day.

Here is the function library.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194

 
# ---------------------------------------------------------------------------
### <Library name='Muegge_LogParser_Lib.ps1'>
### <Author>David Muegge</Author>
### <CreateDate>20081108</CreateDate>
### <ModifiedDate>20081121</ModifiedDate>
### <Description>
### 	Log Parser function library
### </Description>
### <Dependencies>
###		Log Parser 2.2 COM component
### </Dependencies>
### <Usage>
### 	Dot source from calling script
### </Usage>
### </Library>
# ---------------------------------------------------------------------------
 
# ---------------------------------------------------------------------------
### <Function name='Get-LPInputFormat'>
### <Description>
### 	Returns Log Parser Input Format object based on passed string
### </Description>
### <Usage>
###		Get-LPInputFormat -InputType <string>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Get-LPInputFormat{
 
	param([String]$InputType)
 
	switch($InputType.ToLower()){
		"ads"{$inputobj = New-Object -comObject MSUtil.LogQuery.ADSInputFormat}
		"bin"{$inputobj = New-Object -comObject MSUtil.LogQuery.IISBINInputFormat}
		"csv"{$inputobj = New-Object -comObject MSUtil.LogQuery.CSVInputFormat}
		"etw"{$inputobj = New-Object -comObject MSUtil.LogQuery.ETWInputFormat}
		"evt"{$inputobj = New-Object -comObject MSUtil.LogQuery.EventLogInputFormat}
		"fs"{$inputobj = New-Object -comObject MSUtil.LogQuery.FileSystemInputFormat}
		"httperr"{$inputobj = New-Object -comObject MSUtil.LogQuery.HttpErrorInputFormat}
		"iis"{$inputobj = New-Object -comObject MSUtil.LogQuery.IISIISInputFormat}
		"iisodbc"{$inputobj = New-Object -comObject MSUtil.LogQuery.IISODBCInputFormat}
		"ncsa"{$inputobj = New-Object -comObject MSUtil.LogQuery.IISNCSAInputFormat}
		"netmon"{$inputobj = New-Object -comObject MSUtil.LogQuery.NetMonInputFormat}
		"reg"{$inputobj = New-Object -comObject MSUtil.LogQuery.RegistryInputFormat}
		"textline"{$inputobj = New-Object -comObject MSUtil.LogQuery.TextLineInputFormat}
		"textword"{$inputobj = New-Object -comObject MSUtil.LogQuery.TextWordInputFormat}
		"tsv"{$inputobj = New-Object -comObject MSUtil.LogQuery.TSVInputFormat}
		"urlscan"{$inputobj = New-Object -comObject MSUtil.LogQuery.URLScanLogInputFormat}
		"w3c"{$inputobj = New-Object -comObject MSUtil.LogQuery.W3CInputFormat}
		"xml"{$inputobj = New-Object -comObject MSUtil.LogQuery.XMLInputFormat}
 
	}
 
	return $inputobj
 
}
 
 
# ---------------------------------------------------------------------------
### <Function name='Get-LPOutputFormat'>
### <Description>
### 	Returns Log Parser Output Format object based on passed string
### </Description>
### <Usage>
###		Get-LPOutputFormat -OutputType <string>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Get-LPOutputFormat{
 
	param([String]$OutputType)
 
	switch($OutputType.ToLower()){
		"csv"{$outputobj = New-Object -comObject MSUtil.LogQuery.CSVOutputFormat}
		"chart"{$outputobj = New-Object -comObject MSUtil.LogQuery.ChartOutputFormat}
		"iis"{$outputobj = New-Object -comObject MSUtil.LogQuery.IISOutputFormat}
		"sql"{$outputobj = New-Object -comObject MSUtil.LogQuery.SQLOutputFormat}
		"syslog"{$outputobj = New-Object -comObject MSUtil.LogQuery.SYSLOGOutputFormat}
		"tsv"{$outputobj = New-Object -comObject MSUtil.LogQuery.TSVOutputFormat}
		"w3c"{$outputobj = New-Object -comObject MSUtil.LogQuery.W3COutputFormat}
		"tpl"{$outputobj = New-Object -comObject MSUtil.LogQuery.TemplateOutputFormat}
 
	}
 
	return $outputobj
 
}
 
 
# ---------------------------------------------------------------------------
### <Function name='Invoke-LPExecute'>
### <Description>
### 	Executes a Log Parser Query and returns a recordset
### </Description>
### <Usage>
###		Invoke-LPExecute -query <string>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Invoke-LPExecute{
 
	param([string] $query, $inputtype)
 
    $LPQuery = new-object -com MSUtil.LogQuery
	if($inputtype){
    	$LPRecordSet = $LPQuery.Execute($query, $inputtype)	
	}
	else
	{
		$LPRecordSet = $LPQuery.Execute($query)
	}
    return $LPRecordSet
 
}
 
 
# ---------------------------------------------------------------------------
### <Function name='Invoke-LPExecuteBatch'>
### <Description>
### 	Executes Log Parser batch query with passed input and output types
### </Description>
### <Usage>
###		Invoke-LPExecuteBatch -query <string> -inputtype <LogParserInputFormat> -outputtype <LogParserOutputFormat>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Invoke-LPExecuteBatch{
 
	param([string]$query, $inputtype, $outputtype)
 
    $LPQuery = new-object -com MSUtil.LogQuery
    $result = $LPQuery.ExecuteBatch($query, $inputtype, $outputtype)
    return $result
}
 
 
# ---------------------------------------------------------------------------
### <Function name='Get-LPRecord'>
### <Description>
###		Returns PowerShell custom object from Log Parser recordset for current record
### </Description>
### <Usage>
###		Get-LPRecord -rs <RecordSet>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Get-LPRecord{
 
	param($LPRecordSet)
 
	$LPRecord = new-Object System.Management.Automation.PSObject
	if( -not $LPRecordSet.atEnd())
	{
		$Record = $LPRecordSet.getRecord()
		for($i = 0; $i -lt $LPRecordSet.getColumnCount();$i++)
		{        
			$LPRecord | add-member NoteProperty $LPRecordSet.getColumnName($i) -value $Record.getValue($i)
		}
	}
	return $LPRecord
}
 
 
 
# ---------------------------------------------------------------------------
### <Function name='Get-LPRecordSet'>
### <Description>
### 	Executes a Log Parser Query and returns a LogRecordSet as a custom powershell object
### </Description>
### <Usage>
###		Get-LPRecordSet -query <string>
### </Usage>
### </Function>
# ---------------------------------------------------------------------------
function Get-LPRecordSet{
 
	param([string]$query)
 
	# Execute Query
	$LPRecordSet = Invoke-LPExecute $query
	$LPRecords = new-object System.Management.Automation.PSObject[] 0
	for(; -not $LPRecordSet.atEnd(); $LPRecordSet.moveNext())
	{
		# Add record
		$LPRecord = Get-LPRecord($LPRecordSet)
		$LPRecords += new-Object System.Management.Automation.PSObject	
        $RecordCount = $LPQueryResult.length-1
        $LPRecords[$RecordCount] = $LPRecord
	}
	$LPRecordSet.Close();
	return $LPRecords
 
}

This library provides the basic functionality needed for Log Parser. We can use it in two basic scenarios.

One - We want to execute a Log Parser batch. This mode works exactly as the Log Parser command line tool works it queries an input file of a given type and writes the results to an output file of a given type.

Two - We want the results of a Log Parser query returned to a PowerShell object. This will allow us to further process the results using PowerShell or simply utilize the output facilities to display the results.

Example of Scenario One: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

 
. .\Muegge_LogParser_Lib.ps1
 
$infile = "D:\Temp\LPTest.log"
$outfile = "D:\Temp\LPTest.gif"
 
# Set input format and parameters
$inputformat = Get-LPInputFormat "w3c"
 
# Set output format and parameters
$outputformat = Get-LPOutputFormat "chart"
$outputformat.chartTitle = "Top 10 Pages"
$outputformat.chartType = "BarStacked"
$outputformat.groupSize = "600x400"
 
$query = @"
SELECT Top 10 cs-uri-stem, Count(*)
INTO $outfile
FROM $infile
Group By cs-uri-stem
Order by cs-uri-stem desc	
"@
 
# Execute batch
Invoke-LPExecuteBatch $query $inputformat $outputformat | Out-Null

The code above provides a good way to create scheduled reports. The syntax is easier to follow than the command line switches at least for me.

Here is the sample output chart.

Yes, I took this log from a box with nothing but hacker traffic J

Now let’s look at the second scenario. We will return the results of a Log Parser query in a PSObject.

Here we are querying for application hang events in the application event log. We will use a Log Parser query to retrieve just the events we want. Then we will use PowerShell to filter out just the events for IE. The we can easily display the output any way we like. 

1
2
3
4
5
6
7
8
9
10
11
12

 
. .\Muegge_LogParser_Lib.ps1
 
$query = @"
SELECT *
FROM Application
WHERE EventID = 1002
"@
 
$inputformat = Get-LPInputFormat "evt"
$records = Get-LPRecordSet $query $inputformat
$records | where {$_.Strings -like "*iexplore.exe*"} | ft -property TimeGenerated,SourceName,Strings

Here is the output.

TimeGenerated				SourceName   			Strings
-------------     				----------       			-------
11/1/2008 11:21:02 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/2/2008 6:25:35 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/4/2008 7:32:35 AM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/6/2008 4:21:22 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/8/2008 7:05:20 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/10/2008 3:01:45 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/11/2008 5:10:40 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/14/2008 12:13:06 PM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...
11/17/2008 7:34:29 AM			Application Hang			iexplore.exe|7.0.6000.16735|hungapp|0.0...

 

To me this really seems like the best of both worlds utilizing each tool for its strength.

Now I just have to figure out why IE hangs J

Best Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=65 Log Parser and PowerShell – Part I http://muegge.com/blog/?p=62 http://muegge.com/blog/?p=62#comments Wed, 12 Nov 2008 13:33:16 +0000 dmuegge http://muegge.com/blog/?p=62 Welcome!

Log Parser and PowerShell are both great tools and they work well when used together. Yes, you can do pretty much everything Log Parser does with PowerShell alone, but part of PowerShell’s mission is to better leverage current tools. I believe this is an excellent example. Also it has been my experience Log Parser performs better at the task. Steve Schofield also blogged about the performance of Log Parser and PowerShell for querying logs here.

There are two ways to interact with Log Parser from PowerShell. The first and the easiest to get started with is the command line version logparser.exe.

Here is the same command line query example from the last post but with PowerShell.

& ./logparser.exe “SELECT Top 10 cs-uri-stem, Count(*) FROM D:\Logs\ex081110.log Group By cs-uri-stem Order by cs-uri-stem desc” –i:w3c

Big difference isn’t it. J

Here is a simple powershell script using Log Parser. This is a little easier to follow than the batch file example in the last post.

1
2
3
4
5
6
7
8
9
10

$infile = "D:\Temp\LPTest.log"
 
$query = @"
SELECT Top 10 cs-uri-stem, Count(*)
FROM $infile
Group By cs-uri-stem
Order by cs-uri-stem desc	
"@
 
$result = & C:\LogParser\logparser.exe $query -i:w3c

The drawback to these examples is we are not really gaining the full benefit from PowerShell. We could write some functions in PowerShell and convert the text output from Log Parser into PowerShell objects, but there is an easier way to do this.

OK, now for the second way to interact with Log Parser from PowerShell, the Log Parser COM component (logparser.dll). This component installs with Log Parser and should be registered and ready to use if you have installed Log Parser.

The COM component exposes a simple object model consisting of only three main objects. The LogQuery, LogRecorset, and LogRecord objects. There are also a series of objects for the input and output formats.

The COM component is more difficult to use, but is an advantage because data can be returned in object form. We will look at this in detail starting in Log Parser and PowerShell - Part II

Best Regards,

Dave

]]> http://muegge.com/blog/?feed=rss2&p=62