SqlDev PowerShell 102: Working with Data and Variables

In my previous post, SqlDev PowerShell 101: Getting Started with PowerShell, I walked through how to configure and use PowerShell to run basic commands. This post will cover working with data and variables in PowerShell. Specifically, I’m going to demonstrate how to connect to a database (i.e. SQL Server) using Invoke-Sqlcmd, retrieve data, store the data in a variable, and work with that variable.

Note: Invoke-Sqlcmd does require the use of the sqlps module. If you don’t have it installed yet, check here for more information.

From a new PowerShell window, write the following command:

Invoke-Sqlcmd -Query "SELECT DB_NAME();"

PowerShell Working with Data: PowerShell_102_01

Tip: You can auto-complete “Invoke-Sqlcmd” after typing “Invoke-S” and hitting TAB on your keyboard.

What’s happening here? Well, because no database instance was specified, the cmdlet is connecting to the database on the local computer and running the query specified. “Column1″ is returned just as it would be in SQL Server Management Studio (SSMS), because no column name was specified. You can see this by running:

Invoke-Sqlcmd -Query "SELECT DB_NAME() AS MyDatabaseName;"

PowerShell Working with Data: PowerShell_102_02

But what if you don’t have a local database instance, or you want to connect to another database? No problem. Just add the -ServerInstance parameter:

Invoke-Sqlcmd -Query "SELECT DB_NAME() AS MyDatabaseName;" -ServerInstance "."

“.” refers to the local, default SQL Server instance. If you want to connect to a named or remote instance, just replace this value with your server name.

Tip: Hit the “Up” arrow to retrieve the previous results, then just edit the command.

PowerShell Working with Data: PowerShell_102_03

So that’s all pretty easy so far, but honestly, if all I needed to do was run a simple query, I’d do it in SSMS. No, the real power of PowerShell is the .NET Framework. So let’s see this in action by assigning our results to a variable. A variable is defined using the dollar sign ($) and a name; defining a type is optional. Let’s do that now.

$DbName = Invoke-Sqlcmd -Query "SELECT DB_NAME() AS MyDatabaseName;" -ServerInstance "."

PowerShell Working with Data: PowerShell_102_04

In order to view the value stored in the variable, just append a new command to the previous line using a semicolon (;) and the variable’s name.

$DbName = Invoke-Sqlcmd -Query "SELECT DB_NAME() AS MyDatabaseName;" -ServerInstance "."; $DbName

PowerShell Working with Data: PowerShell_102_05

Optionally, you could also type the variable’s name and press Enter.

$DbName

PowerShell Working with Data: PowerShell_102_06

Now let’s run a query that returns multiple columns:

$DbDetails = Invoke-Sqlcmd -Query "SELECT database_id, name, state_desc, recovery_model_desc FROM sys.databases WHERE name = 'master';" -ServerInstance "."; $DbDetails

PowerShell Working with Data: PowerShell_102_07

We didn’t define a data type, so let’s see what data type PowerShell assigned for us. To do this, we’ll use the GetType method:

$DbDetails.GetType()

PowerShell Working with Data: PowerShell_102_08

You can see that the variable is defined as a system object; specifically, it’s a data row. That’s important, because it determines how we’ll interact with the variable. For instance, what if we wanted to reference the value stored in the variable, not the variable as an object? You can do this by referencing the column stored in the object directly, i.e.

$DbDetails.name

PowerShell Working with Data: PowerShell_102_09

Now let’s run the same query, but remove the database criteria from the WHERE statement:

$DbDetailsAll = Invoke-Sqlcmd -Query "SELECT database_id, name, state_desc, recovery_model_desc FROM sys.databases;" -ServerInstance "."; $DbDetailsAll

PowerShell Working with Data: PowerShell_102_10

Your results will vary, of course, but the important thing is that you should have more than 1 row returned.

Now let’s return just the database names again:

$DbDetailsAll.name

PowerShell Working with Data: PowerShell_102_11

Hopefully you’re starting to see how retrieving an array of values and accessing just a single attribute could be useful.

Let’s now iterate through our list of databases and perform an action. In this example, we’ll just iterate through each database and retrieve the count of tables. To do this, we’ll need to use a Foreach loop.

Foreach ($DatabaseRow In $DbDetailsAll) {$TableCount = Invoke-Sqlcmd -Query "SELECT COUNT(*) AS TableCount FROM sys.tables;" -ServerInstance "." -Database $DatabaseRow.name; $TableCount }

PowerShell Working with Data: PowerShell_102_12

This is the biggest statement we’ve written yet, so let’s take a minute to break it down.

We’re starting the loop by calling Foreach, then we’re specifying how we’re iterating through the array. As each row in the array comes into focus, we’re storing the row in the $DatabaseRow variable that we’ve just defined. Everything specified inside the curly brackets ({}) is executing for each row in the array. We’re then connecting to the local instance and changing the context of the database to be the value currently stored in $DatabaseRow. The results are stored in a new variable, $TableCount, which is then outputted to the screen.

It’s actually a pretty short piece of code, given all that it’s doing. But it’s difficult to understand which TableCount maps to which table, so let’s add that information to our output:

Foreach ($DatabaseRow In $DbDetailsAll) {$TableCount = Invoke-Sqlcmd -Query "SELECT '$DatabaseRow.name' AS DatabaseName, COUNT(*) AS TableCount FROM sys.tables;" -ServerInstance "." -Database $DatabaseRow.name; $TableCount }

PowerShell Working with Data: PowerShell_102_13

Woah, that’s not what we want! PowerShell is displaying the object type, not the value. In order to actually retrieve the value from an object variable, we’re going to store the database name in a string variable first, then reference that newly-defined variable.

Foreach ($DatabaseRow In $DbDetailsAll) {$DatabaseName = $DatabaseRow.name; $TableCount = Invoke-Sqlcmd -Query "SELECT '$DatabaseName' AS DatabaseName, COUNT(*) AS TableCount FROM sys.tables;" -ServerInstance "." -Database $DatabaseRow.name; $TableCount }

PowerShell Working with Data: PowerShell_102_14

Voila! Now we have a list of databases on a server and the associated count of tables in each database.

If you’re wondering why you’d ever actually use this PowerShell script, well… you probably wouldn’t. But it’s important to have these building blocks before going into more advanced concepts. In my next post, we’ll be creating a config table of SQL Server instances, connecting to each server, retrieving details about missing indexes, and writing those results out to disk. That should be a little more useful. ;)

Until then… happy scripting! :)

SqlDev PowerShell 101: Getting Started with PowerShell

This is the first in a new series focused on PowerShell for SQL developers. That means using PowerShell to create and/or modify table schema, stored procedures, and indexes. In other words, automating code development and database tuning processes. This is mostly what I’ve been using PowerShell for so far, and I have to say, I am LOVING it! It’s allowing me to save *days* of coding and validating.

Disclaimer: I am NOT a PowerShell expert, just a newly converted enthusiast who wants to save you from the headaches I myself have suffered while trying to learn PoSh. So I’ll mostly focus on the “how to’s” instead of the “why’s” in this series. :)

I’m going to start at the very beginning and assume you have never opened PowerShell before in your life. Basically, I’m going to assume you’re where I was just a few short months ago. So let us start by finding the application. ;)

If you’re using Windows 7 or later, the good news is that PowerShell comes pre-installed on the OS. We just need to configure it to allow the execution of scripts. Click on the Start icon –> “Search programs and files” –> type “PowerShell.”

Searching for PowerShell

Searching for PowerShell

The first time we open PowerShell, we will need to run it as Administrator for elevated permissions. To do this, right-click on the Windows PowerShell icon that should now be on your Taskbar, then click on “Run as Administrator.” While you’re there, also add it to your Taskbar and/or Start Menu for easy retrieval in the future.

Running PowerShell as Administrator

Running PowerShell as Administrator

Now we need to change the Execution Policy from “Restricted” — which prevents the execution of PowerShell scripts — to something a little more accessible. For the purposes of this post, we’re going to set it to “RemoteSigned,” with the understanding that this may pose some security risks. You can read more about Execution Policy settings here: Using the Get-ExecutionPolicy Cmdlet.

Similarly, you can run this command inside of PowerShell to get more details:

Get-Help About_Signing

Now that we have PowerShell open in Administrator mode, execute this command:

Set-ExecutionPolicy RemoteSigned

You will be prompted to confirm the change. Type “Y”. You can then confirm that the change has taken place by typing:

Get-ExecutionPolicy
Changing Execution Policy

Changing Execution Policy

Alrighty, now we’re ready to write some PowerShell!

Go ahead and close your PowerShell window, then re-open as you normally would (not in Administrator mode; we don’t need elevated permissions any longer).

Let’s start with some simple commands. For this post, I have scripted all objects from AdventureWorks to a new directory called C:\MyCode\. We can take a look at the files in the directory using a cmdshell command that’s probably very familiar: Dir. Go ahead and run that command now.

Dir
Running Dir in PowerShell

Running Dir in PowerShell

So does that mean PowerShell supports all the old cmdshell commands that we know and love (to hate)? Well, yes and no. The PowerShell devs, forward-thinking folks that they are, knew that having support for cmdshell syntax would probably ease the transition for many people, so they created aliases. You can see this by running the Get-Alias cmdlet:

Get-Alias Dir
Get-Alias Dir

Get-Alias Dir

What we see is that, under the covers, Dir is nothing more than an alias for Get-ChildItem. So let’s switch to using Get-ChildItem from here on out.

One thing that took me a while to discover, but which is very helpful, is the auto-complete functionality inherent in PowerShell. To use this, just start typing a command, such as “Get-Ch”, and hit the Tab key. As you press Tab, PowerShell will iterate through the list of available cmdlets / switches that are available in the current context. Keep hitting Tab until you get to Get-ChildItem, then execute that command.

Get-ChildItem
Get-ChildItem

Get-ChildItem

Hmm, those results look familiar. That’s a good sign. But what else can we do with Get-ChildItem? Let’s find out by using the Get-Help cmdlet:

Get-Help Get-ChildItem
Get-Help Get-ChildItem

Get-Help Get-ChildItem

Type “Get-ChildItem -” (make sure to include the hyphen, or you’ll be tabbing through way more than you’d like) and press Tab to iterate through the list of available switches. Stop when you get to the -Recurse switch. If you go too far, don’t worry… you can hit the Shift+Tab to go in reverse direction. This allows Get-ChildItem to execute against child folders too, so we’ll add that to our command.

Get-ChildItem -Recurse

So far, so good. Now what else can we do?

At this point, we need to discuss piping. Piping is a way to stack multiple cmdlets together to produce a single, final, perfect result. It’s one of the features that adds the “power” in PowerShell. You can read more about piping here: Piping and the Pipeline in Windows PowerShell.

So let’s build our first pipe. We know how to return all the files in a directory, but how do we *search* the directory for files that contain a specific keyword? It turns out it’s actually not that hard… we just need to use the Select-String cmdlet. To do this, press the Up arrow on your keyboard to retrieve the previously executed command, then type a pipe (|) before adding the Select-String cmdlet:

Get-ChildItem -Recurse | Select-String -Pattern "SalesOrderDetail"
Search Files

Search Files

So what’s this doing? Well, it’s iterating through each of the files listed in the directory and subdirectories, then calling the Select-String cmdlet to find files that contain the word “SalesOrderDetail.” It’s even showing us what line the pattern is found on. Pretty awesome, huh? But I think we can do better.

One of the best things about PowerShell is the sheer amount of documentation available. Case in point: TechNet has a quick reference of PowerShell 2.0 cmdlets. Looking at the list, some interesting cmdlets catch my eye, including Select-Object and Get-Unique. If we pipe these cmdlets to our existing command, we get something like this:

Get-ChildItem -Recurse | Select-String -Pattern "SalesOrderDetail" | Sort-Object | Select-Object Path | Get-Unique -AsString
Final PowerShell Command

Final PowerShell Command

This returns a very easy-to-consume list of files that contain the pattern “SalesOrderDetail.” Imagine using this to search through hundreds of files and thousands of lines of code to find dependencies. Powerful PowerShell indeed.

So that’s it for this first post and introduction to PowerShell. If you’re looking for more resources, here are some PowerShell-related sites I’ve found especially helpful:

Happy Scripting! :)

Conversion Issues Upgrading to PowerShell 3.0

This post probably has a small audience, but there were a handful of conversion issues I ran into when I upgraded to PowerShell 3.0. It was difficult finding posts related to the errors I encountered, so I’m posting my experiences in the hopes that it helps someone else down the road.

The first issue I encountered was calling PowerShell from xp_cmdshell on a remote SQL Server.

In PowerShell 2.0, I used this syntax:

EXEC xp_cmdshell 'powershell.exe -Command "get-executionpolicy"'
# PoSh 2.0 Results
AllSigned
 
# PoSh 3.0 Results
'powershell.exe' is not recognized as an internal or external command,
operable program or batch file.

To fix this issue, I had to fully qualify the PowerShell directory:

EXEC xp_cmdshell 'c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command "get-executionpolicy"'
# PoSh 3.0 Results
AllSigned

Another issue I ran into was calling the Teradata .NET Data Provider. In PowerShell 2.0, I was calling the Invoke-SqlStmt function I previously posted to connect from a Windows server to a remote Teradata server.

PS X:\> .\Invoke-SqlStmt.ps1 -C TDATA -S MyTDServerName -Q "SELECT * FROM DBC.Tables;" -Verbose

Everything worked great. However, once I upgraded to PowerShell 3.0, I was getting this error:

Exception calling "GetFactory" with "1" argument(s): "Unable to find the requested .Net Framework Data Provider.  It may not be installed."
At line:1 char:1
+ $Factory = [System.Data.Common.DbProviderFactories]::GetFactory("Teradata.Client ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentException

I couldn’t find anything that would explain this behavior, and I confirmed that everything still worked great when ran from PowerShell 2.0. Ultimately, I decided to install the Teradata .NET Data Provider locally, which solved the problem. I didn’t find anything that explained why I could previously use a remote adapter and now can only use a local adapter. I don’t really understand the problem, but I was able to solve it.

Here’s the syntax to install the Teradata .NET Data Provider remotely using xp_cmdshell.

EXEC xp_cmdshell 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command "X:\drop\tdnetdp__windows_indep.13.11.00.01.exe /s /v"/qn"'

Note that this code is installing the 13.11 adapter, so you may need to change the filename if you’re installing a different version.

And to leave things on a positive note, one awesome improvement of PowerShell 3.0 over 2.0 is the redirection operators. I’m a big fan of logging and verbose messaging, but in 2.0, I had to put extra steps into the code to log these messages. In 3.0, I’ve been able to log my verbose output simply by appending “*> directory” to my commands, i.e.

EXEC xp_cmdshell 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command "X:\MyAwesomeScript.ps1 -Verbose *> X:\MyAwesomeLog.txt"'

I think that is just so cool. You can read more about the different redirection operators by running this in PowerShell:

get-help redirect

Happy Scripting! :)

Invoke-SqlStmt – PowerShell for SQL Server & Teradata Queries

I’ve been using PowerShell quite a bit the last few months, and I finally feel I’m to the point where I can start sharing some of what I’ve learned.

I started a blog post that walks through using PowerShell for database development and automation, and I quickly realized that a lot of what I want to show is dependent upon the first function I wrote, Invoke-SqlStmt. And by “wrote,” I mean “blatantly borrowed from Chad Miller’s Invoke-SqlCmd2 script;) . Thank you, Chad! The main difference is that my function supports connecting to either Teradata or SQL Server, depending on the parameters passed.

One thing worth mentioning is that I spent a lot of time researching all of the different ways to connect to SQL Server. In the end, I went with ADO.NET, as a lot of what I will be doing is working with data sets (i.e. running a query against ServerA and writing the results to ServerB).

Invoke-SqlStmt

##################################################################################################
#
#   .NAME
#       Invoke-SqlStmt.ps1
#
#   .SYNOPSIS
#       Executes a SQL statement against either a Teradata or SQL Server source using ADO.NET
#
#   .INPUT
#       C - Connection Type; Required; MSSQL or TDATA
#       S - Server Name; Required; server name for any valid SQL Server or data instance
#       Q - Query; Optional; either Q or F is required
#       F - Filename; Optional; either Q or F is required
#       U - Username; Optional; if not supplied, Integrated mode is assumed
#       P - Password; Optional; if not supplied, Integrated mode is assumed
#       T - Connection Timeout; Optional; defaults to 600s but can be overridden
#       AS - Optional; can specify whether DataSet, DataTable, or DataRow will be returned
#
#   .OUTPUTS
#       System.Data.(DataSet|DataTable|DataRow)
#
#   .CREDIT
#       Modified from Chad Miller's Invoke-SqlCmd2 script:
#       http://gallery.technet.microsoft.com/scriptcenter/7985b7ef-ed89-4dfd-b02a-433cc4e30894#content
#
#  --------------------------------------------------------------------------------------------
#
#   .EXAMPLES
#
#   Invoke-SqlStmt -C MSSQL -S "(local)" -Q "SELECT @@SERVERNAME;"
#     This example connects to a named instance of the D Engine on a computer and runs a basic T-SQL Q.
#
#   Invoke-SqlStmt -C MSSQL -S "(local)" -F "C:\myScript.sql" | Out-File -filePath "C:\myOutput.txt"
#     This example reads a file containing T-SQL statements, runs the file, and writes the output to another file.
#
#   Invoke-SqlStmt  -C MSSQL -S "(local)" -Q "PRINT 'hello world'" -Verbose
#     This example uses the PowerShell -Verbose parameter to return the message output of the PRINT command.
#
#   Invoke-SqlStmt  -C TDATA -S "(local)" -Q "SELECT * FROM DBC.Tables;"
#     This example connects to Teradata and executes a query. 
#
#  --------------------------------------------------------------------------------------------
#
#   AUTHOR:     Michelle Ufford
#   CREATED:    2012-12-12
#
#  CHANGE HISTORY:
#  --------------------------------------------------------------------------------------------
#  ChangeDate/Version   Author                  Description
#  --------------------------------------------------------------------------------------------
#  20121212.0           Michelle Ufford         Initial Creation
#  20121218.1           Michelle Ufford         Tweaks
##################################################################################################
 
function Invoke-SqlStmt
{
    [CmdletBinding()]
    param(
    [Parameter(Position=0, Mandatory=$false)] [ValidateSet("MSSQL", "TDATA")] [string]$C="MSSQL",
    [Parameter(Position=1, Mandatory=$true)]  [string]$S,
    [Parameter(Position=2, Mandatory=$false)] [string]$D,
    [Parameter(Position=3, Mandatory=$false)] [string]$Q,
    [Parameter(Position=4, Mandatory=$false)] [ValidateScript({test-path $_})] [string]$F,
    [Parameter(Position=5, Mandatory=$false)] [string]$U,
    [Parameter(Position=6, Mandatory=$false)] [string]$P,
    [Parameter(Position=8, Mandatory=$false)] [Int32]$T=600,
    [Parameter(Position=9, Mandatory=$false)] [ValidateSet("DataSet", "DataTable", "DataRow")] [string]$AS="DataRow"
    )
 
    # Check for input files
    if ($F)
    {
        $filePath = $(resolve-path $F).path
        $Q =  [System.IO.File]::ReadAllText("$filePath")
    }
 
    # ----------------------
    # Teradata Connections
    # ----------------------
    if ($C -eq "TDATA")
    {
        $ConnectionString = "Data Source={0}; User Id={1}; Password={2}; Connection Pooling Timeout={3};" -f $S, $U, $P, $T
 
        Add-Type -AssemblyName System.Data
        $Factory = [System.Data.Common.DbProviderFactories]::GetFactory("Teradata.Client.Provider")
        $Connection = $Factory.CreateConnection()
        $Connection.ConnectionString = $ConnectionString
        $Connection.Open()
        $Command = $Connection.CreateCommand()
        $Command.CommandText = $Q
        $DataSet = New-Object System.Data.DataSet
        $DataAdapter = $Factory.CreateDataAdapter()
        $DataAdapter.SelectCommand = $Command
        $DataAdapter.SelectCommand.CommandTimeout = $T
        [void] $DataAdapter.Fill($DataSet) | Out-Null
        $Connection.Close()
 
        # get a table contained in the DataSet, http://technet.microsoft.com/en-us/library/dd347701.aspx
        switch ($As)
        {
              'DataSet'   { Write-Output ($DataSet) }
              'DataTable' { Write-Output ($DataSet.Tables) }
              'DataRow'   { Write-Output ($DataSet.Tables[0]) }
        }
    }
    else
    # ------------------------
    # SQL Server Connections
    # ------------------------
    {
        $Connection = New-Object System.Data.SqlClient.SQLConnection
 
        if ($U)
        { $ConnectionString = "Server={0};Database={1};User ID={2};Password={3};Trusted_Connection=False;Connect Timeout={4}" -f $S,$D,$U,$P,$T }
        else
        { $ConnectionString = "Server={0};Database={1};Integrated Security=True;Connect Timeout={2}" -f $S,$D,$CT }
 
        $Connection.ConnectionString=$ConnectionString
 
        #Following EventHandler is used for PRINT and RAISERROR T-SQL statements. Executed when -Verbose parameter specified by caller
        if ($PSBoundParameters.Verbose)
        {
            $Connection.FireInfoMessageEventOnUserErrors=$true
            $Handler = [System.Data.SqlClient.SqlInfoMessageEventHandler] {Write-Verbose "$($_)"}
            $Connection.add_InfoMessage($Handler)
        }
 
        $Connection.Open()
        $Command = New-Object System.Data.SqlClient.SqlCommand
        $Command.Connection = $Connection
        $Command.CommandText = $Q
        $Command.CommandTimeout = $T
        $DataSet = New-Object System.Data.DataSet
        $DataAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
        $DataAdapter.SelectCommand = $Command
        [void]$DataAdapter.Fill($DataSet) | Out-Null
        $Connection.Close()
 
        switch ($As)
        {
            'DataSet'   { Write-Output ($DataSet) }
            'DataTable' { Write-Output ($DataSet.Tables) }
            'DataRow'   { Write-Output ($DataSet.Tables[0]) }
        }
    }
}

I plan to have a follow-up post up shortly that will walk you through how to use this script step-by-step.