How to Monitor ADO.NET Connection Pools on Multiple SQL Servers with PowerShell

When using ADO.NET to make SQL Server database connections, connection pooling is used to minimize the cost of repeatedly opening and closing a new database connection. For a full explanation of connection pooling, see SQL Server Connection Pooling (ADO.NET)

The default maximum size of a connection pool is 100. If a connection is not properly closed/disposed in application code, it can remain in the connection pool unused until garbage collection cleans it up manually. The garbage collection process can take several minutes before closing an open connection. In a busy application with many database calls, the connection pool default max of 100 can be exhausted quickly if connections are not closed/disposed properly. When this happens, a typical exception would be: “Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.”  

I recently encountered a scenario like this with extensive and complex application code that made connections to multiple SQL Servers and databases, so tracking down the improper connection string in the application code was proving difficult and time consuming. To aid in the troubleshooting, I created the PowerShell script below that monitors connection pool size and usage by host & host process id in order to narrow down the source of improper connection string(s).

To execute the script, two input files are necessary. First, create a “Servers.txt” file containing all the SQL Servers that your applications(s) make connections to. The Servers.txt file should be saved in the same directory as the PowerShell script and should look like this (Just replace “Server#” with your real server names.):

Servers

Secondly create a “FunctionLibrary.ps1” file containing the Invoke-SQLCmd2 and Write-DataTable custom functions. For more on adding these two functions and their use, see a full and thorough write up at the Hey, Scripting Guy! Blog. (Note that I’ve leveraged some of the concepts from this blog post into my own.) This FunctionLibrary.ps1 file should also be saved in the same directory as the other files listed in this post.

Finally, save the PowerShell Script below as a .ps1 file. (In the example shown here, I’ve named the file MonitorConnectionPools.ps1) Just replace YOUR_SERVER and YOUR_DATABASE with a real server name and database in your environment. A great place for this table would be on a DBA utility server or some other non-production server you use for administrative tasks.

#Add Functions
. ./FunctionLibrary.ps1</pre>
$ErrorActionPreference = "Stop"
$ServerList = Get-Content Servers.txt

$Query = "DECLARE @CurrentDate DATETIME = GETDATE();
SELECT  @@SERVERNAME AS 'ServerName', login_name, Host_name, host_process_id, COUNT(*) AS 'ConnectionCount', @CurrentDate AS 'CollectionTime'
FROM sys.dm_exec_sessions 
WHERE host_process_id IS NOT NULL
GROUP BY Host_name, login_name, host_process_id
HAVING COUNT(*) >= 25;"

foreach ($Server in $ServerList)
{
$Results = Invoke-Sqlcmd2 -Server $Server -Database master -Query $Query -As 'DataTable'

Write-DataTable -ServerInstance "YOUR_SERVER" -Database "YOUR_DATABASE" -TableName "ConnectionPoolMonitor" -Data $Results
}

 

Now that the PowerShell setup is in place, the next step is to create a table to store the results. This table should be created on the server you specified for YOUR_SERVER  in the database specified as YOUR_DATABASE in the MonitorConnectionPools.ps1 script above. To create the table structure, run the following SQL script:


SELECT  @@SERVERNAME AS 'ServerName'
,login_name
,host_name
,host_process_id
,COUNT(*) AS 'ConnectionCount'
,GETDATE() AS 'CollectionTime'
INTO [YOUR_DATABASE].dbo.[ConnectionPoolMonitor]
FROM sys.dm_exec_sessions 
WHERE host_process_id = -1
GROUP BY Host_name, login_name, host_process_id

Note that there is not a host_process_id of “-1”, so the SELECT returns no rows. This just creates the table with the desired structure in preparation for the real writes that will take place later.

Lastly, set up a SQL Server Job that calls your newly created MonitorConnectionPools.ps1 script. Set the Job up as follows:

Monitor Connection Pools Job Set up (Fixed)

In the example above, I’ve saved the MonitorConnectionPools.ps1, FunctionLibrary.ps1, and Servers.txt files in a local folder called “C:\Powershell\MonitorConnectionPools“. Use the Set-Location command to navigate to your saved file location.  Schedule the SQL Job to run every minute (or two) and it will write connection counts per host & host_process_id to the ConnectionPoolMonitor table created above. The script only collects connections in excess of 25 to avoid excess noise in the table. You can adjust the threshold as needed in the MonitorConnectionPools.ps1 file.

Once the SQL Job has been running for some time, review the data collected by querying the ConnectionPoolMonitor table. Look specifically for any instances of “100”  (or close to it) in ConnectionCount. If any connection pool timeout errors are seen in conjunction with the 100 connections, there’s likely an ADO.NET connection string that is missing a proper close/dispose command.

100 Connections

In the example above, host process 4732 using the login name “App_Login” running from the server “App_Host” is consuming all 100 of the connections available in the connection pool. If connection pool timeouts are seen during the time period associated with the CollectionTime above, this would be the culprit.

To determine exactly what the host process is (in this case, host process 4732), run the Get-Process -Id 4732 PowerShell command on the App_Host server. If the process is an IIS Worker Process (w3wp.exe), one more step is necessary to determine the correct application pool that is causing the issue. From the Command Line on the App_Host server, navigate to the inetsrv folder and execute appcmd list wps to get the name of each worker process:

appcmd list wps

Names obfuscated to protect the innocent. 🙂

This will show you the name of the application pool with the improperly closed/disposed connection so it can be easily located (and hopefully fixed!) in the application code.

This post is full of a lot of information, but hopefully it will help you on your way to monitoring and troubleshooting ADO.NET connection pooling with SQL Server!

How to Disable/Enable SQL Alerts on Multiple SQL Servers with PowerShell

When managing SQL Server AlwaysOn Availability Groups and/or standard Database Mirroring, setting up SQL Alerts should be standard protocol. (See this SQL Server Pro article for more background on this.) These types of Alerts can be configured to send e-mails any time the defined event occurs which is great for being notified of unexpected failovers and errors. However, the mass of e-mails generated during a regular planned maintenance that involves one or more failovers can be a little annoying. (Or maybe that’s just me… But if you’re reading this, maybe that’s just you as well. 🙂 I work with dozens of servers that have multiple Availability Groups and Mirrored Databases across them. Whenever we perform planned maintenance, the first thing I do is to disable all Mirroring and Availability Group related Alerts to spare my Inbox from the onslaught of unnecessary SQL Alert e-mails notifying me of the failovers. To do this quickly and efficiently, I’ve written a PowerShell script to disable/enable multiple Alerts on any number of servers automatically.

To execute the script, two input files are necessary. First, create a “Alert_Servers.txt” file containing all the servers with Alerts to be disabled. The Alert_Servers.txt file should be saved in the same directory as the PowerShell script and should look like this (Just replace “Server#” with your real server names.):

Alert Servers

Secondly create a “Alerts.txt” file containing the specific Alerts you want to disable/enable. This file should also be saved in the same directory as the PowerShell script and should look similar to this:

Alerts Text

In the example above, I’ve added all the specific Mirroring and Availability Group Alerts we set up as standard on all our servers. You can get a list of all configured Alerts on your servers by simply running this query:

SELECT name FROM msdb.dbo.sysalerts;

Just add new lines to this file for each Alert you’d like to disable/enable.

Finally, save the PowerShell Script below as a .ps1 file. (In the example shown below, I’ve named the file ToggleAlerts.ps1)

if (!(Get-PSSnapin -Name SQLServerCmdletSnapin100 -ErrorAction SilentlyContinue)) {
Add-PSSnapin SQLServerCmdletSnapin100}
if (!(Get-PSSnapin -Name SqlServerProviderSnapin100 -ErrorAction SilentlyContinue)) {
Add-PSSnapin SqlServerProviderSnapin100}

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") | Out-Null
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo.Extended") | Out-Null

$Enabled = read-host "Enable or Disable? Please enter '1' or '0'"

if ($Enabled.ToString() -ne 1 -and $Enabled.ToString() -ne 0)
{
Write-Output "Only 1 and 0 are valid entries."
Exit
}

$ServerList = Get-Content Alert_Servers.txt
$AlertList  = Get-Content Alerts.txt

Write-Output "Please wait..."

foreach ($Server in $ServerList)
{
foreach ($Alert in $AlertList)
{
$Query = "IF EXISTS (SELECT 1 FROM msdb..sysalerts WHERE name = '$Alert')
EXEC msdb.dbo.sp_update_alert
@name = '$Alert'
,@enabled = $Enabled"

Invoke-Sqlcmd -Server $Server -Database "msdb" -Query $Query
}
}

if ($Enabled -eq 1)
{Write-Output "Alerts Enabled!"}

if ($Enabled -eq 0)
{Write-Output "Alerts Disabled!"}

 

Once you’ve created the ToggleAlerts.ps1 and the two text files, open a PowerShell window and execute the script. In the example below, I’ve saved all three files to the folder “C:\PowerShell\ToggleAlerts”. Enter “0” to disable alerts prior to the start of maintenance:

AlertsPS1

Once your failovers and maintenance are complete, run the same script again but enter “1” instead to re-enable the Alerts:

AlertsPS2

That’s all it takes to save your Inbox during a regular maintenance window!

How to Filter and Review SQL Error Logs on Multiple SQL Servers with PowerShell

When working with many related SQL Servers it can be helpful to review SQL Error Logs on multiple servers all at once while also filtering for certain types of error messages. There are many use cases for this script, but the most common use case I’ve encountered is to check for failed logins and connection timeouts after an Availability Group or Mirrored Database failover. I work in an environment with multiple Always On Availability Groups with dozens of webservers that connect to them via Listeners. When we do regular maintenance such as Windows Patching, we manually failover our Availability Groups as part of the patching process. With one run of this PowerShell script, we can ensure there are no failed logins or connection issues once the failovers and maintenance have been completed. Some of our Availability Groups span up to five Replicas, so there have been occasions where our Logins are out of sync.

To execute the script, two input files are necessary. First, create a “Servers.txt” file containing all the servers with error logs to review. The Servers.txt file should be saved in the same directory as the PowerShell script and should look like this (Just replace “Server#” with your real server names.):

Servers

Secondly create a “Error_Filter.txt” file containing the specific error messages you’re interested in searching for. This file should also be saved in the same directory as the PowerShell script and should look similar to this:

Error Entries

In the example above, we’re looking for any error message containing “login failed” or “connection timeout”.  Any number of different filter criteria can be included in the search by simply adding a new line to this file.

Finally, save the PowerShell Script below as a .ps1 file. (In the example shown below, I’ve named the file CheckErrorLogs.ps1)

if (!(Get-PSSnapin -Name SQLServerCmdletSnapin100 -ErrorAction SilentlyContinue)) {
Add-PSSnapin SQLServerCmdletSnapin100}
if (!(Get-PSSnapin -Name SqlServerProviderSnapin100 -ErrorAction SilentlyContinue)) {
Add-PSSnapin SqlServerProviderSnapin100}

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo") | Out-Null
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo.Extended") | Out-Null

$DateLimit = read-host "Please enter a minimum date/time in the following format:
[YYYY-MM-DD HH:MM]   i.e. 2015-01-01 15:30
Leave blank to omit the date time filter.

Enter Date"

$ServerList = Get-Content Servers.txt
$ErrorList  = Get-Content Error_Filter.txt
$Results=@()

foreach ($Server in $ServerList)
{
foreach ($Error in $ErrorList)
{
$Query = "SET NOCOUNT ON
CREATE TABLE #ERRORLOG (LogDate DATETIME, ProcessInfo VARCHAR(64), [Text] VARCHAR(MAX))
INSERT INTO #ERRORLOG
exec sp_readerrorlog
SELECT @@SERVERNAME AS 'ServerName', LogDate, [Text] FROM #ERRORLOG
WHERE [Text] LIKE '$Error'
AND LogDate > '$DateLimit'
DROP TABLE #ERRORLOG"

$Results += Invoke-Sqlcmd -Server $Server -Database "master" -Query $Query
}
}

$Results | Out-GridView

Once you’ve created the CheckErrorLogs.ps1 and the two text files, open a PowerShell window and execute the script. In the example below, I’ve saved all three files to the folder “C:\PowerShell\ReadSQLErrorLogs”.

Call CheckErrorLogs Small2

Enter a minimum date to filter for recent error messages only, or leave it blank to get all error messages in the current error log.

Enter Date Small

Finally, the results are output to a grid view for easy review and sorting.

Error Log Results w blank boxes Small2

Hopefully, this will save you a lot of time and effort when you need to review multiple SQL error logs!

How to Check Instant File Initialization and Lock Pages in Memory on All Your SQL Servers with PowerShell

For performance reasons, it is generally recommended that the Windows Service Account used by SQL Server be given permissions to Perform Volume Maintenance Tasks (i.e. Instant File Initialization) and to Lock Pages in Memory. (For more on why these permissions should be granted, see Jes Schultz Borland’s Will Instant File Initialization Really Help My Databases? and Jonathan Kehayias’ Great SQL Server Debates: Lock Pages in Memory.)

I work in an environment with dozens of production SQL Servers with seasonal peak workloads occurring twice yearly. One of my team’s tasks prior to each peak season is to confirm that each of our production SQL Servers have these specific permissions enabled. This can be accomplished by logging on manually to each server and reviewing the User Rights Assignments of the Local Security Policy. But who has the time for that?? (Or, this could be a great job for the intern… 🙂 To accomplish this task efficiently via automation, I’ve written a PowerShell script that will iterate through a list of servers and return the requested Security Policy information.

To execute the script, you’ll need to create a “Servers.txt” file containing all the servers to check. The Servers.txt file should be saved in the same directory as the PowerShell script and should look like this (Just replace “Server#” with your real server names.):

Servers

Next, save the PowerShell Script below as a .ps1 file. (In the example shown below, I’ve named the file ValidatePermissions.ps1)

$ServerList = Get-Content Servers.txt

foreach ($Server in $ServerList)
{

Invoke-Command -ComputerName $Server -ScriptBlock {

Set-Location C:\

$LocalSvr = get-content env:computername

$proc = Get-CimInstance Win32_Process -Filter “name = 'sqlservr.exe'”
$CimMethod = Invoke-CimMethod -InputObject $proc -MethodName GetOwner

$objUser = New-Object System.Security.Principal.NTAccount($CimMethod.Domain, $CimMethod.User)
$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
$NTName = $strSID.Value

$ManageVolumePriv = 0
$LockPagesPriv = 0

secedit /export /areas USER_RIGHTS /cfg UserRights.inf /quiet

$FileResults = Get-Content UserRights.inf

Remove-Item UserRights.inf

foreach ($line in $FileResults)
{
if($line -like "SeManageVolumePrivilege*" -and $line -like "*$NTName*")
{
$ManageVolumePriv = 1
}

if($line -like "SeLockMemoryPrivilege*" -and $line -like "*$NTName*")
{
$LockPagesPriv = 1
}
}

Write-Host "Server:" $LocalSvr -foregroundcolor black -backgroundcolor gray
Write-Host " Lock Pages In Memory:" $LockPagesPriv
Write-Host " Instant File Initialization:" $ManageVolumePriv

}

}
 

Once you’ve created the ValidatePermissions.ps1 file and the Servers.txt file, open a PowerShell window and execute the script. In the example below, I’ve saved both files to the folder “C:\PowerShell”.

PowerShellResult

If any of the SQL Servers checked were lacking permissions, a “0” would return instead of a “1“. Rather than spending 20 minutes checking these servers manually, we’ve now confirmed the appropriate permissions in seconds! This script can be run anytime as needed with no further configuration.

Hopefully, this timesaver will allow you more of an opportunity to work on your 9-iron!