SQL Fool » SQL 2008

Index Interrogation for SQL Server 2008

Michelle Ufford — Fri, 04 Jun 2010 19:55:17 +0000

I had previously posted an index interrogation script for SQL Server 2005. I've updated that script for 2008; namely, it includes filtered index definitions. For anyone interested:

DECLARE @objectID INT = OBJECT_ID('Sales.SalesOrderHeader');
 
WITH indexCTE(partition_scheme_name
            , partition_function_name
            , data_space_id)
AS (
    SELECT sps.name
        , spf.name
        , sps.data_space_id
    FROM sys.partition_schemes AS sps
    Join sys.partition_functions AS spf
        ON sps.function_id = spf.function_id
)
 
SELECT st.name AS 'table_name'
    , IsNull(ix.name, '') AS 'index_name'
    , ix.OBJECT_ID
    , ix.index_id
	, CAST(
        CASE WHEN ix.index_id = 1 
                THEN 'clustered' 
            WHEN ix.index_id =0
                THEN 'heap'
            ELSE 'nonclustered' END
		+ CASE WHEN ix.ignore_dup_key <> 0 
            THEN ', ignore duplicate keys' 
                ELSE '' END
		+ CASE WHEN ix.is_unique <> 0 
            THEN ', unique' 
                ELSE '' END
		+ CASE WHEN ix.is_primary_key <> 0 
            THEN ', primary key' ELSE '' END AS VARCHAR(210)
        ) AS 'index_description'
    , IsNull(REPLACE( REPLACE( REPLACE(
        (   
            SELECT c.name AS 'columnName'
            FROM sys.index_columns AS sic
            Join sys.columns AS c 
                ON c.column_id = sic.column_id 
                And c.OBJECT_ID = sic.OBJECT_ID
            WHERE sic.OBJECT_ID = ix.OBJECT_ID
                And sic.index_id = ix.index_id
                And is_included_column = 0
            ORDER BY sic.index_column_id
            FOR XML Raw)
            , '"/>', ')
            , ''')
            , '"/>', ''), '')
        AS 'indexed_columns'
    , IsNull(REPLACE( REPLACE( REPLACE(
        (   
            SELECT c.name AS 'columnName'
            FROM sys.index_columns AS sic
            Join sys.columns AS c 
                ON c.column_id = sic.column_id 
                And c.OBJECT_ID = sic.OBJECT_ID
            WHERE sic.OBJECT_ID = ix.OBJECT_ID
                And sic.index_id = ix.index_id
                And is_included_column = 1
            ORDER BY sic.index_column_id
            FOR XML Raw)
            , '"/>', ')
            , ''')
            , '"/>', ''), '')
        AS 'included_columns'
    , ix.filter_definition
    , IsNull(cte.partition_scheme_name, '') AS 'partition_scheme_name'
    , COUNT(partition_number) AS 'partition_count'
    , SUM(ROWS) AS 'row_count'
FROM sys.indexes AS ix
Join sys.partitions AS sp
    ON ix.OBJECT_ID = sp.OBJECT_ID
    And ix.index_id = sp.index_id
Join sys.tables AS st
    ON ix.OBJECT_ID = st.OBJECT_ID
LEFT Join indexCTE AS cte
    ON ix.data_space_id = cte.data_space_id
WHERE ix.OBJECT_ID = IsNull(@objectID, ix.OBJECT_ID)
GROUP BY st.name
    , IsNull(ix.name, '')
    , ix.OBJECT_ID
    , ix.index_id
	, CAST(
        CASE WHEN ix.index_id = 1 
                THEN 'clustered' 
            WHEN ix.index_id =0
                THEN 'heap'
            ELSE 'nonclustered' END
		+ CASE WHEN ix.ignore_dup_key <> 0 
            THEN ', ignore duplicate keys' 
                ELSE '' END
		+ CASE WHEN ix.is_unique <> 0 
            THEN ', unique' 
                ELSE '' END
		+ CASE WHEN ix.is_primary_key <> 0 
            THEN ', primary key' ELSE '' END AS VARCHAR(210)
        )
    , ix.filter_definition
    , IsNull(cte.partition_scheme_name, '')
    , IsNull(cte.partition_function_name, '')
ORDER BY table_name
    , index_id;

You may need to create some indexes to see this in AdventureWorks:

CREATE NONCLUSTERED INDEX IX_Sales_SalesOrderHeader_filtered_2005
    ON Sales.SalesOrderHeader(AccountNumber)
    Include (CustomerID, SalesPersonID)
    WHERE OrderDate >= '2005-01-01'
        And OrderDate < '2006-01-01';

table_name           index_name                               object_id   index_id    index_description                   indexed_columns      included_columns               filter_definition                                            partition_scheme_name partition_count row_count
-------------------- ---------------------------------------- ----------- ----------- ----------------------------------- -------------------- ------------------------------ ------------------------------------------------------------ --------------------- --------------- --------------------
SalesOrderHeader     PK_SalesOrderHeader_SalesOrderID         1010102639  1           clustered, unique, primary key      SalesOrderID                                        NULL                                                                               1               31465
SalesOrderHeader     AK_SalesOrderHeader_rowguid              1010102639  2           nonclustered, unique                rowguid                                             NULL                                                                               1               31465
SalesOrderHeader     AK_SalesOrderHeader_SalesOrderNumber     1010102639  3           nonclustered, unique                SalesOrderNumber                                    NULL                                                                               1               31465
SalesOrderHeader     IX_SalesOrderHeader_CustomerID           1010102639  5           nonclustered                        CustomerID                                          NULL                                                                               1               31465
SalesOrderHeader     IX_SalesOrderHeader_SalesPersonID        1010102639  6           nonclustered                        SalesPersonID                                       NULL                                                                               1               31465
SalesOrderHeader     IX_Sales_SalesOrderHeader_filtered_2005  1010102639  13          nonclustered                        AccountNumber        CustomerID, SalesPersonID      ([OrderDate]>='2005-01-01' AND [OrderDate]<'2006-01-01')                           1               1379

Filtered Indexes Work-Around

Michelle Ufford — Mon, 01 Feb 2010 16:00:14 +0000

Recently, I needed to create a stored procedure that queried a rather large table. The table has a filtered index on a date column, and it covers the query. However, the Query Optimizer was not using the index, which was increasing the execution time (not to mention IO!) by at least 10x. This wasn't the first time I've had the Optimizer fail to use a filtered index. Normally when this happens, I use a table hint to force the filtered index -- after I verify that it is indeed faster, of course. However, since this was a stored procedure, I was receiving the following error message whenever I tried to execute the proc:

Query processor could not produce a query plan because of the hints defined in this query. Resubmit the query without specifying any hints and without using SET FORCEPLAN.

SQL Server would not allow me to execute the stored procedure using the filtered index hint. If I removed the hint, it executed, but it used a different, non-covering and far more expensive index. For those of you not familiar with this issue, allow me to illustrate the problem.

First, create a table to play with and populate it with some bogus data:

CREATE TABLE dbo.filteredIndexTest
(
      myID   INT IDENTITY(1,3)
    , myDate SMALLDATETIME 
    , myData CHAR(100)
 
    CONSTRAINT PK_filteredIndexTest
        PRIMARY KEY CLUSTERED(myID)
);
 
SET NOCOUNT ON;
DECLARE @DATE SMALLDATETIME = '2010-01-01';
 
WHILE @DATE < '2010-02-01'
BEGIN
 
    INSERT INTO dbo.filteredIndexTest
    (
          myDate
        , myData
    )
    SELECT @DATE
        , 'Date: ' + CONVERT(VARCHAR(20), @DATE, 102);
 
    SET @DATE = DATEADD(MINUTE, 1, @DATE);
 
END;
 
SELECT COUNT(*) FROM dbo.filteredIndexTest;

It looks like this will generate 44,640 rows of test data... plenty enough for our purposes. Now, let's create our filtered index and write a query that will use it:

CREATE NONCLUSTERED INDEX IX_filteredIndexTest_1
    ON dbo.filteredIndexTest(myDate)
    Include (myData)
    WHERE myDate >= '2010-01-27';
 
SELECT DISTINCT myData
FROM dbo.filteredIndexTest
WHERE myDate >= '2010-01-28';

If you look at the execution plan for this query, you'll notice that the Optimizer is using the filtered index. Perfect! Now let's parameterize it.

DECLARE @myDate1 SMALLDATETIME = '2010-01-28';
 
SELECT DISTINCT myData
FROM dbo.filteredIndexTest
WHERE myDate >= @myDate1;

Uh oh. Looking at the execution plan, we see that SQL Server is no longer using the filtered index. Instead, it's scanning the clustered index! Why is this? There's actually a good explanation for it. The reason is that I could, in theory, pass a date to my parameter that fell outside of the filtered date range. If that's the case, then SQL Server could not utilize the filtered index. Personally, I think it's a bug and SQL Server should identify whether or not a filtered index could be used based on the actual value submitted, but... that's a whole other blog post.

So what can we do? Well, dynamic SQL may be able to help us out in this case. Let's give it a go. First, let's try parameterized dynamic SQL.

DECLARE @mySQL1 NVARCHAR(2000)
    , @myParam NVARCHAR(2000) = '@p_myDate2 smalldatetime'
    , @myDate2 SMALLDATETIME = '2010-01-28';
 
SET @mySQL1 = 'Select Distinct myData
              From dbo.filteredIndexTest
              Where myDate >= @p_myDate2';
 
EXECUTE SP_EXECUTESQL @mySQL1, @myParam, @p_myDate2 = @myDate2;

Looking at the execution plan, we see we're still scanning on the clustered index. This is because the parameterized dynamic SQL resolves to be the exact same query as the one above it. Let's try unparameterized SQL instead:

DECLARE @mySQL2 NVARCHAR(2000)
    , @myDate3 SMALLDATETIME = '2010-01-28';
 
SET @mySQL2 = 'Select Distinct myData
              From dbo.filteredIndexTest
              Where myDate >= ''' + CAST(@myDate3 AS VARCHAR(20)) + '''';
 
EXECUTE SP_EXECUTESQL @mySQL2;
 
-- Drop Table dbo.filteredIndexTest;

Voila! We have a seek on our filtered index. Why? Because the statement resolves to be identical to our first query, where we hard-coded the date value in the WHERE clause.

Now, I want to stress this fact: you should always, ALWAYS use parameterized dynamic SQL whenever possible. Not only is it safer, but it's also faster, because it can reuse cached plans. But sometimes you just cannot accomplish the same tasks with it. This is one of those times. If you do end up needing to use unparameterized dynamic SQL as a work-around, please make sure you're validating your input, especially if you're interfacing with any sort of external source.

There's an even easier work-around for this problem that Dave (http://www.crappycoding.com) shared with me: recompile.

Adding "Option (Recompile)" to the end of your statements will force the Optimizer to re-evaluate which index will best meet the needs of your query every time the statement is executed. More importantly, it evaluates the plan based on the actual values passed to the parameter... just like in our hard-coded and dynamic SQL examples. Let's see it in action:

DECLARE @myDate4 SMALLDATETIME = '2010-01-28';
 
SELECT DISTINCT myData
FROM dbo.filteredIndexTest 
WHERE myDate >= @myDate4
OPTION (RECOMPILE);
 
DECLARE @myDate5 SMALLDATETIME = '2010-01-20';
 
SELECT DISTINCT myData
FROM dbo.filteredIndexTest 
WHERE myDate >= @myDate5
OPTION (RECOMPILE);

If we look at the execution plans for the 2 queries above, we see that the first query seeks on the filtered index, and the second query scans on the clustered index. This is because the second query cannot be satisfied with the filtered index because we initially limited our index to dates greater than or equal to 1/27/2010.

There are, of course, trade-offs associated with each approach, so use whichever one best meets your needs. Do you have another work-around for this issue? If so, please let me know.

Update:

Alex Kuznetsov (http://www.simple-talk.com/author/alex-kuznetsov/) shared this method too:

DECLARE @myDate1 SMALLDATETIME = '2010-01-28';
 
SELECT DISTINCT myData
FROM dbo.filteredIndexTest
WHERE myDate = @myDate1
AND myDate >= '2010-01-27';

Like the other examples, this will result in an index seek on the filtered index. Basically, by explicitly declaring the start date of your filter, you're letting the Optimizer know that the filtered index can satisfy the request, regardless of the parameter value passed. Thanks for the tip, Alex!

Index Defrag Script Updates – Beta Testers Needed

Michelle Ufford — Tue, 26 Jan 2010 21:15:37 +0000

Update: Wow! I've received a ton of responses to my request for beta testers. Thank you all! The SQL Community is really amazing. I'll hopefully have the new version online in just a few days.

Over the last few months, I've received many great comments and suggestions regarding my Index Defrag Script v3.0. I've just recently had time to implement most of these suggestions, plus some other things that I thought would be useful.

Here's some of what you can look forward to shortly:

Probably the single most requested feature, the new version of the script allows you to set a time limit for index defrags.
There's now a static table for managing the status of index defrags. This way, when your time limit is reached, you can pick up where you left off the next day, without the need to rescan indexes.
There's now an option to prioritize defrags by range scan counts, fragmentation level, or page counts.
For those using partitioning, there is now an option to exclude the right-most populated partition from defrags (in theory, the one you're writing to in a sliding-window scenario).
Options such as page count limits and SORT_IN_TEMPDB are now parameterized.
I've enhanced error logging.
... and more!

Right now, I'm looking for a few folks who are willing to beta test the script. If you're interested, please send me an e-mail at michelle at sqlfool dot com with the editions of SQL Server you can test this on (i.e. 2005 Standard, 2008 Enterprise, etc.).

Thank you!

Partitioning Tricks

Michelle Ufford — Fri, 09 Oct 2009 19:12:35 +0000

For those of you who are using partitioning, or who are considering using partitioning, allow me to share some tips with you.

Easy Partition Staging Tables

Switching partitions (or more specifically, hobts) in and out of a partitioned table requires the use of a staging table. The staging table has very specific requirements: it must be completely identical to the partitioned table, including indexing structures, and it must have a check constraint that limits data to the partitioning range. Thanks to my co-worker Jeff, I've recently started using the SQL Server Partition Management tool on CodePlex. I haven't used the automatic partition switching feature -- frankly, using any sort of data modification tool in a production environment makes me nervous -- but I've been using the scripting option to create staging tables in my development environment, which I then copy to production for use. It's nothing you can't do yourself, but it does make the whole process easy and painless, plus it saves you from annoying typos. But be careful when using this tool to just create the table and check constraints automatically, because you may need to...

Add Check Constraints After Loading Data

Most of the time, I add the check constraint when I create the staging table, then I load data and perform the partition switch. However, for some reason, I was receiving the following error:

.Net SqlClient Data Provider: Msg 4972, Level 16, State 1, Line 1
ALTER TABLE SWITCH statement failed. Check constraints or partition function of source table 'myStagingTable' allows values that are not allowed by check constraints or partition function on target table 'myDestinationTable'.

This drove me crazy. I confirmed my check constraints were correct, that I had the correct partition number, and that all schema and indexes matched identically. After about 30 minutes of this, I decided to drop and recreate the constraint. For some reason, it fixed the issue. Repeat tests produced the same results: the check constraint needed to be added *after* data was loaded. This error is occurring on a SQL Server 2008 SP1 box; to be honest, I'm not sure what's causing the error, so if you know, please leave me a comment. But I figured I'd share so that anyone else running into this issue can hopefully save some time and headache.

Replicating Into Partitioned and Non-Partitioned Tables

Recently, we needed to replicate a non-partitioned table to two different destinations. We wanted to use partitioning for Server A, which has 2008 Enterprise; Server B, which is on 2005 Standard, could not take advantage of partitioning. The solution was really easy: create a pre-snapshot and post-snapshot script for the publication, then modify to handle each server group differently. Using pseudo-code, it looked something like this:

/* Identify which servers get the partitioned version */
IF @@SERVERNAME In ('yourServerNameList') 
BEGIN
 
    /* Create your partitioning scheme if necessary */
    IF Not Exists(SELECT * FROM sys.partition_schemes WHERE name = 'InsertPartitionScheme')
        CREATE PARTITION SCHEME InsertPartitionScheme 
            AS PARTITION InsertPartitionFunction ALL TO ([PRIMARY]);    
 
    /* Create your partitioning function if necessary */
    IF Not Exists(SELECT * FROM sys.partition_functions WHERE name = 'InsertPartitionFunction')
        CREATE PARTITION FUNCTION InsertPartitionFunction (SMALLDATETIME) 
            AS RANGE RIGHT FOR VALUES ('insertValues');    
 
    /* Create a partitioned version of your table */
    CREATE TABLE [dbo].[yourTableName] (
	    [yourTableSchema]
    ) ON InsertPartitionScheme([partitioningKey]);
 
END
ELSE
BEGIN
 
    /* Create a non-partitioned version of your table */
    CREATE TABLE [dbo].[yourTableName] (
	    [yourTableSchema]
    ) ON [PRIMARY];
 
END

You could also use an edition check instead of a server name check, if you prefer. The post-snapshot script basically looked the same, except you create partitioned indexes instead.

Compress Old Partitions

Did you know you can set different compression levels for individual partitions? It's true! I've just completed doing this on our largest partitioned table. Here's how:

/* Apply compression to your partitioned table */
ALTER TABLE dbo.yourTableName
Rebuild Partition = All
WITH 
(
      Data_Compression = Page ON Partitions(1 TO 9)
    , Data_Compression = ROW  ON Partitions(10 TO 11) 
    , Data_Compression = NONE ON Partitions(12)
);
 
/* Apply compression to your partitioned index */
ALTER INDEX YourPartitionedIndex
    ON dbo.yourTableName
    Rebuild Partition = All
    WITH 
    (
      Data_Compression = Page ON Partitions(1 TO 9)
    , Data_Compression = ROW  ON Partitions(10 TO 11) 
    , Data_Compression = NONE ON Partitions(12)
    );
 
/* Apply compression to your unpartitioned index */
ALTER INDEX YourUnpartitionedIndex
    ON dbo.yourTableName
    Rebuild WITH (Data_Compression = ROW);

A couple of things to note. In all of our proof-of-concept testing, we found that compression significantly reduced query execution time, reads (IO), and storage. However, CPU was also increased significantly. The results were more dramatic, both good and bad, with page compression versus row compression. Still, for our older partitions, which aren't queried regularly, it made sense to turn on page compression. The newer partitions receive row compression, and the newest partitions, which are still queried very regularly by routine processes, were left completely uncompressed. This seems to strike a nice balance in our environment, but of course, results will vary depending on how you use your data.

Something to be aware of is that compressing your clustered index does *not* compress your non-clustered indexes; those are separate operations. Lastly, for those who are curious, it took us about 1 minute to apply row compression and about 7 minutes to apply page compression to partitions averaging 30 million rows.

Looking for more information on table partitioning? Check out my overview of partitioning, my example code, and my article on indexing on partitioned tables.

Undocumented Function in SQL 2008

Michelle Ufford — Wed, 02 Sep 2009 01:39:52 +0000

If you've been following my blog for a little while, you'll know that I'm a fan of SQL Server internals. There's a lot that can be learned or better understood by rolling up your sleeves and getting into the nitty-gritty of data pages (i.e. see my post on Overhead in Non-Unique Clustered Indexes). So imagine how happy I was when my co-worker Jeff shared an undocumented function with me today that retrieves the file number, page number, and slot number of a single record. Very cool! Well, at least to me. So now let's see how you can use it.

The fn_physLocCracker function can be called in the following way:

SELECT TOP 100 plc.*, soh.SalesOrderID
FROM Sales.SalesOrderHeader AS soh
Cross Apply sys.fn_physLocCracker (%%physloc%%) AS plc;

Results (just a sample):

file_id     page_id     slot_id     SalesOrderID
----------- ----------- ----------- ------------
1           14032       0           43659
1           14032       1           43660
1           14032       2           43661
1           14032       3           43662
1           14032       4           43663

If you look at the sp_helptext for sys.fn_physLocCracker, %%physloc%% is apparently a virtual column that contains information on where the record is stored. In fact, you can even append %%physloc%% to your column list if you want to see how the information is stored. But for our purposes, we now have a file number, page number, and slot number. What do we do with it?

Well, you can use the investigation proc I wrote to retrieve the actual data page:

EXECUTE dba_viewPageData_sp 
      @databaseName = 'AdventureWorks'
    , @fileNumber = 1
    , @pageNumber = 14032;

Results (just a sample):

Slot 0 Column 1 Offset 0x4 Length 4 Length (physical) 4
 
SalesOrderID = 43659                 
 
Slot 0 Column 2 Offset 0x8 Length 1 Length (physical) 1
 
RevisionNumber = 1                   
 
Slot 0 Column 3 Offset 0x9 Length 8 Length (physical) 8
 
OrderDate = 2001-07-01 00:00:00.000  
 
Slot 0 Column 4 Offset 0x11 Length 8 Length (physical) 8
 
DueDate = 2001-07-13 00:00:00.000

Neat, huh? So why would you use it to look up the data page and file number when you can just pass the table name and index name to my proc and retrieve data pages? Well, my investigation proc will retrieve data pages for any index type -- the fn_physLocCracker function will only retrieve data for the clustered index -- but it will not retrieve the data page for a specific record. So just something to be aware of.

That's all for now. Back to the #24HoursOfPASS!

Index Defrag Script, v3.0

Michelle Ufford — Wed, 24 Jun 2009 00:37:24 +0000

UPDATE: This script has been significantly updated. Find the latest version here: Index Defrag Script, v4.0

I've just completed the latest version of my index defrag script! Here's a brief list of the updates:

Fixed a bug with the LOB logic. In the previous version, after a LOB was encountered, all subsequent indexes would be reorganized.
Added support for stat rebuilds after the index defrag is complete (@rebuildStats)
Added an exclusion list table (dba_indexDefragExclusion) to support index scheduling
Modified logging to show which defrags are "in progress"; added columns to dba_indexDefragLog
Added support for the defrag of the model and msdb databases
Added @scanMode as a configurable parameter

So what can this index defrag script do? Well, for starters, you can:

Schedule it to run with the default settings; it works "right out of the box" with no additional configuration necessary
Run this one script from a centralized database for all databases on a server
Run this script for a specific database or table
Configure custom threshold limits and the point at which a rebuild should be performed (instead of a reorganize)
Defrag individual partitions
Log its actions and the duration of the defrag
Run in "commands only" mode (@executeSQL = 0, @printCommands = 1)
Customize performance parameters such as @maxDopRestriction and @defragDelay to minimize impact on the server
Schedule specific indexes to only be defragged on weekends, or every other day

To use this last option, you need to add a record to the dba_indexDefragExclusion table. I think all of the columns are pretty self-explanatory except the [exclusionMask] column. The way this works is each day of the week is assigned a value:
1=Sunday, 2=Monday, 4=Tuesday, 8=Wednesday, 16=Thursday, 32=Friday, 64=Saturday

Take a SUM of the values for the days that you want excluded. So if you want an index to only be defragged on weekends, you would add up Monday through Friday (2+4+8+16+32) and use a value of 62 for the exclusionMask column. For a little more information on how this works, check out my blog post on Bitwise Operations.

Please note: if you don't insert any records into the dba_indexDefragExclusion table, by default all indexes will be defragged every run-time if they exceed the specified thresholds. This is normal behavior and may be perfectly fine in your environment. However, if the dba_indexDefragExclusion table does not exist, the script will fail.

I try to document each parameter within the code, so check the comments section in the script for a full list of parameters and what they do.

Special thanks to everyone who helped beta test this script!

Without further ado, the script:

/* Drop Table Scripts:
Drop Table dbo.dba_indexDefragLog;
Drop Table dbo.dba_indexDefragExclusion;
*/
IF Not Exists(SELECT [OBJECT_ID] FROM sys.tables 
    WHERE [name] In (N'dba_indexDefragLog', 'dba_indexDefragExclusion'))
BEGIN
 
    CREATE TABLE dbo.dba_indexDefragLog
    (
          indexDefrag_id    INT IDENTITY(1,1)   Not Null
        , databaseID        INT                 Not Null
        , databaseName      NVARCHAR(128)       Not Null
        , objectID          INT                 Not Null
        , objectName        NVARCHAR(128)       Not Null
        , indexID           INT                 Not Null
        , indexName         NVARCHAR(128)       Not Null
        , partitionNumber   SMALLINT            Not Null
        , fragmentation     FLOAT               Not Null
        , page_count        INT                 Not Null
        , dateTimeStart     DATETIME            Not Null
        , dateTimeEnd       DATETIME            Null
        , durationSeconds   INT                 Null
 
        CONSTRAINT PK_indexDefragLog 
            PRIMARY KEY CLUSTERED (indexDefrag_id)
    );
 
    PRINT 'dba_indexDefragLog Table Created';
 
    CREATE TABLE dbo.dba_indexDefragExclusion
    (
          databaseID        INT                 Not Null
        , databaseName      NVARCHAR(128)       Not Null
        , objectID          INT                 Not Null
        , objectName        NVARCHAR(128)       Not Null
        , indexID           INT                 Not Null
        , indexName         NVARCHAR(128)       Not Null
        , exclusionMask     INT                 Not Null
            /* 1=Sunday, 2=Monday, 4=Tuesday, 8=Wednesday, 16=Thursday, 32=Friday, 64=Saturday */
 
        CONSTRAINT PK_indexDefragExclusion 
            PRIMARY KEY CLUSTERED (databaseID, objectID, indexID)
    );
 
    PRINT 'dba_indexDefragExclusion Table Created';
 
END
ELSE
    RAISERROR('One or more tables already exist.  Please drop or rename before proceeding.', 16, 0);
 
IF OBJECTPROPERTY(OBJECT_ID('dbo.dba_indexDefrag_sp'), N'IsProcedure') = 1
BEGIN
    DROP PROCEDURE dbo.dba_indexDefrag_sp;
    PRINT 'Procedure dba_indexDefrag_sp dropped';
END;
Go
 
 
CREATE PROCEDURE dbo.dba_indexDefrag_sp
 
    /* Declare Parameters */
      @minFragmentation     FLOAT           = 5.0  
        /* in percent, will not defrag if fragmentation less than specified */
    , @rebuildThreshold     FLOAT           = 30.0  
        /* in percent, greater than @rebuildThreshold will result in rebuild instead of reorg */
    , @executeSQL           BIT             = 1     
        /* 1 = execute; 0 = print command only */
    , @DATABASE             VARCHAR(128)    = Null
        /* Option to specify a database name; null will return all */
    , @tableName            VARCHAR(4000)   = Null  -- databaseName.schema.tableName
        /* Option to specify a table name; null will return all */
    , @scanMode             VARCHAR(10)     = N'LIMITED'
        /* Options are LIMITED, SAMPLED, and DETAILED */
    , @onlineRebuild        BIT             = 1     
        /* 1 = online rebuild; 0 = offline rebuild; only in Enterprise */
    , @maxDopRestriction    TINYINT         = Null
        /* Option to restrict the number of processors for the operation; only in Enterprise */
    , @printCommands        BIT             = 0     
        /* 1 = print commands; 0 = do not print commands */
    , @printFragmentation   BIT             = 0
        /* 1 = print fragmentation prior to defrag; 
           0 = do not print */
    , @defragDelay          CHAR(8)         = '00:00:05'
        /* time to wait between defrag commands */
    , @debugMode            BIT             = 0
        /* display some useful comments to help determine if/where issues occur */
    , @rebuildStats         BIT             = 1
        /* option to rebuild stats after completed index defrags */
 
AS
/*********************************************************************************
    Name:       dba_indexDefrag_sp
 
    Author:     Michelle Ufford, http://sqlfool.com
 
    Purpose:    Defrags all indexes for the current database
 
    Notes:
 
    CAUTION: TRANSACTION LOG SIZE SHOULD BE MONITORED CLOSELY WHEN DEFRAGMENTING.
 
      @minFragmentation     defaulted to 10%, will not defrag if fragmentation 
                            is less than that
 
      @rebuildThreshold     defaulted to 30% as recommended by Microsoft in BOL;
                            greater than 30% will result in rebuild instead
 
      @executeSQL           1 = execute the SQL generated by this proc; 
                            0 = print command only
 
      @database             Optional, specify specific database name to defrag;
                            If not specified, all non-system databases will
                            be defragged.
 
      @tableName            Specify if you only want to defrag indexes for a 
                            specific table, format = databaseName.schema.tableName;
                            if not specified, all tables will be defragged.
 
      @scanMode             Specifies which scan mode to use to determine
                            fragmentation levels.  Options are:
                            LIMITED - scans the parent level; quickest mode,
                                      recommended for most cases.
                            SAMPLED - samples 1% of all data pages; if less than
                                      10k pages, performs a DETAILED scan.
                            DETAILED - scans all data pages.  Use great care with
                                       this mode, as it can cause performance issues.
 
      @onlineRebuild        1 = online rebuild; 
                            0 = offline rebuild
 
      @maxDopRestriction    Option to specify a processor limit for index rebuilds
 
      @printCommands        1 = print commands to screen; 
                            0 = do not print commands
 
      @printFragmentation   1 = print fragmentation to screen;
                            0 = do not print fragmentation
 
      @defragDelay          Time to wait between defrag commands; gives the
                            server a little time to catch up 
 
      @debugMode            1 = display debug comments; helps with troubleshooting
                            0 = do not display debug comments
 
      @rebuildStats         Affects only statistics that need to be rebuilt
                            1 = rebuild stats
                            0 = do not rebuild stats
 
    Called by:  SQL Agent Job or DBA
 
    Date        Initials	Version Description
    ----------------------------------------------------------------------------
    2007-12-18  MFU         1.0     Initial Release
    2008-10-17  MFU         1.1     Added @defragDelay, CIX_temp_indexDefragList
    2008-11-17  MFU         1.2     Added page_count to log table
                                    , added @printFragmentation option
    2009-03-17  MFU         2.0     Provided support for centralized execution
                                    , consolidated Enterprise & Standard versions
                                    , added @debugMode, @maxDopRestriction
                                    , modified LOB and partition logic  
    2009-06-18  MFU         3.0     Fixed bug in LOB logic, added @scanMode option
                                    , added support for stat rebuilds (@rebuildStats)
                                    , support model and msdb defrag
                                    , added columns to the dba_indexDefragLog table
                                    , modified logging to show "in progress" defrags
                                    , added defrag exclusion list (scheduling)
*********************************************************************************
    Exec dbo.dba_indexDefrag_sp
          @executeSQL           = 0
        , @printCommands        = 1
        , @debugMode            = 1
        , @printFragmentation   = 1;
*********************************************************************************/																
 
SET NOCOUNT ON;
SET XACT_Abort ON;
SET Ansi_Padding ON;
SET Ansi_Warnings ON;
SET ArithAbort ON;
SET Concat_Null_Yields_Null ON;
SET Numeric_RoundAbort OFF;
SET Quoted_Identifier ON;
 
BEGIN
 
    IF @debugMode = 1 RAISERROR('Undusting the cogs and starting up...', 0, 42) WITH NoWait;
 
    /* Declare our variables */
    DECLARE   @objectID             INT
            , @databaseID           INT
            , @databaseName         NVARCHAR(128)
            , @indexID              INT
            , @partitionCount       BIGINT
            , @schemaName           NVARCHAR(128)
            , @objectName           NVARCHAR(128)
            , @indexName            NVARCHAR(128)
            , @partitionNumber      SMALLINT
            , @fragmentation        FLOAT
            , @pageCount            INT
            , @sqlCommand           NVARCHAR(4000)
            , @rebuildCommand       NVARCHAR(200)
            , @dateTimeStart        DATETIME
            , @dateTimeEnd          DATETIME
            , @containsLOB          BIT
            , @editionCheck         BIT
            , @debugMessage         VARCHAR(128)
            , @updateSQL            NVARCHAR(4000)
            , @partitionSQL         NVARCHAR(4000)
            , @partitionSQL_Param   NVARCHAR(1000)
            , @LOB_SQL              NVARCHAR(4000)
            , @LOB_SQL_Param        NVARCHAR(1000)
            , @rebuildStatsID       INT
            , @rebuildStatsSQL      NVARCHAR(1000)
            , @indexDefrag_id       INT;
 
    /* Create our temporary tables */
    CREATE TABLE #indexDefragList
    (
          databaseID        INT
        , databaseName      NVARCHAR(128)
        , objectID          INT
        , indexID           INT
        , partitionNumber   SMALLINT
        , fragmentation     FLOAT
        , page_count        INT
        , defragStatus      BIT
        , schemaName        NVARCHAR(128)   Null
        , objectName        NVARCHAR(128)   Null
        , indexName         NVARCHAR(128)   Null
    );
 
    CREATE TABLE #databaseList
    (
          databaseID        INT
        , databaseName      VARCHAR(128)
        , scanStatus        BIT
        , statsStatus       BIT
    );
 
    CREATE TABLE #processor 
    (
          [INDEX]           INT
        , Name              VARCHAR(128)
        , Internal_Value    INT
        , Character_Value   INT
    );
 
    IF @debugMode = 1 RAISERROR('Beginning validation...', 0, 42) WITH NoWait;
 
    /* Just a little validation... */
    IF @minFragmentation Not Between 0.00 And 100.0
        SET @minFragmentation = 10.0;
 
    IF @rebuildThreshold Not Between 0.00 And 100.0
        SET @rebuildThreshold = 30.0;
 
    IF @defragDelay Not Like '00:[0-5][0-9]:[0-5][0-9]'
        SET @defragDelay = '00:00:05';
 
    IF @scanMode Not In ('LIMITED', 'SAMPLED', 'DETAILED')
        SET @scanMode = 'LIMITED';
 
    /* Make sure we're not exceeding the number of processors we have available */
    INSERT INTO #processor
    EXECUTE XP_MSVER 'ProcessorCount';
 
    IF @maxDopRestriction IS Not Null And @maxDopRestriction > (SELECT Internal_Value FROM #processor)
        SELECT @maxDopRestriction = Internal_Value
        FROM #processor;
 
    /* Check our server version; 1804890536 = Enterprise, 610778273 = Enterprise Evaluation, -2117995310 = Developer */
    IF (SELECT SERVERPROPERTY('EditionID')) In (1804890536, 610778273, -2117995310) 
        SET @editionCheck = 1 -- supports online rebuilds
    ELSE
        SET @editionCheck = 0; -- does not support online rebuilds
 
    IF @debugMode = 1 RAISERROR('Grabbing a list of our databases...', 0, 42) WITH NoWait;
 
    /* Retrieve the list of databases to investigate */
    INSERT INTO #databaseList
    SELECT database_id
        , name
        , 0 -- not scanned yet for fragmentation
        , 0 -- statistics not yet updated
    FROM sys.databases
    WHERE name = IsNull(@DATABASE, name)
        And [name] Not In ('master', 'tempdb')-- exclude system databases
        And [STATE] = 0; -- state must be ONLINE
 
    IF @debugMode = 1 RAISERROR('Looping through our list of databases and checking for fragmentation...', 0, 42) WITH NoWait;
 
    /* Loop through our list of databases */
    WHILE (SELECT COUNT(*) FROM #databaseList WHERE scanStatus = 0) > 0
    BEGIN
 
        SELECT TOP 1 @databaseID = databaseID
        FROM #databaseList
        WHERE scanStatus = 0;
 
        SELECT @debugMessage = '  working on ' + DB_NAME(@databaseID) + '...';
 
        IF @debugMode = 1
            RAISERROR(@debugMessage, 0, 42) WITH NoWait;
 
       /* Determine which indexes to defrag using our user-defined parameters */
        INSERT INTO #indexDefragList
        SELECT
              database_id AS databaseID
            , QUOTENAME(DB_NAME(database_id)) AS 'databaseName'
            , [OBJECT_ID] AS objectID
            , index_id AS indexID
            , partition_number AS partitionNumber
            , avg_fragmentation_in_percent AS fragmentation
            , page_count 
            , 0 AS 'defragStatus' /* 0 = unprocessed, 1 = processed */
            , Null AS 'schemaName'
            , Null AS 'objectName'
            , Null AS 'indexName'
        FROM sys.dm_db_index_physical_stats (@databaseID, OBJECT_ID(@tableName), Null , Null, @scanMode)
        WHERE avg_fragmentation_in_percent >= @minFragmentation 
            And index_id > 0 -- ignore heaps
            And page_count > 8 -- ignore objects with less than 1 extent
            And index_level = 0 -- leaf-level nodes only, supports @scanMode
        OPTION (MaxDop 2);
 
        /* Keep track of which databases have already been scanned */
        UPDATE #databaseList
        SET scanStatus = 1
        WHERE databaseID = @databaseID;
 
    END
 
    CREATE CLUSTERED INDEX CIX_temp_indexDefragList
        ON #indexDefragList(databaseID, objectID, indexID, partitionNumber);
 
    /* Delete any indexes from our to-do that are also in our exclusion list for today */
    DELETE idl
    FROM #indexDefragList AS idl
    Join dbo.dba_indexDefragExclusion AS ide
        ON idl.databaseID = ide.databaseID
        And idl.objectID = ide.objectID
        And idl.indexID = ide.indexID
    WHERE exclusionMask & POWER(2, DATEPART(weekday, GETDATE())-1) > 0;
 
    SELECT @debugMessage = 'Looping through our list... there''s ' + CAST(COUNT(*) AS VARCHAR(10)) + ' indexes to defrag!'
    FROM #indexDefragList;
 
    IF @debugMode = 1 RAISERROR(@debugMessage, 0, 42) WITH NoWait;
 
    /* Begin our loop for defragging */
    WHILE (SELECT COUNT(*) FROM #indexDefragList WHERE defragStatus = 0) > 0
    BEGIN
 
        IF @debugMode = 1 RAISERROR('  Picking an index to beat into shape...', 0, 42) WITH NoWait;
 
        /* Grab the most fragmented index first to defrag */
        SELECT TOP 1 
              @objectID         = objectID
            , @indexID          = indexID
            , @databaseID       = databaseID
            , @databaseName     = databaseName
            , @fragmentation    = fragmentation
            , @partitionNumber  = partitionNumber
            , @pageCount        = page_count
        FROM #indexDefragList
        WHERE defragStatus = 0
        ORDER BY fragmentation DESC;
 
        IF @debugMode = 1 RAISERROR('  Looking up the specifics for our index...', 0, 42) WITH NoWait;
 
        /* Look up index information */
        SELECT @updateSQL = N'Update idl
            Set schemaName = QuoteName(s.name)
                , objectName = QuoteName(o.name)
                , indexName = QuoteName(i.name)
            From #indexDefragList As idl
            Inner Join ' + @databaseName + '.sys.objects As o
                On idl.objectID = o.object_id
            Inner Join ' + @databaseName + '.sys.indexes As i
                On o.object_id = i.object_id
            Inner Join ' + @databaseName + '.sys.schemas As s
                On o.schema_id = s.schema_id
            Where o.object_id = ' + CAST(@objectID AS VARCHAR(10)) + '
                And i.index_id = ' + CAST(@indexID AS VARCHAR(10)) + '
                And i.type > 0
                And idl.databaseID = ' + CAST(@databaseID AS VARCHAR(10));
 
        EXECUTE SP_EXECUTESQL @updateSQL;
 
        /* Grab our object names */
        SELECT @objectName  = objectName
            , @schemaName   = schemaName
            , @indexName    = indexName
        FROM #indexDefragList
        WHERE objectID = @objectID
            And indexID = @indexID
            And databaseID = @databaseID;
 
        IF @debugMode = 1 RAISERROR('  Grabbing the partition count...', 0, 42) WITH NoWait;
 
        /* Determine if the index is partitioned */
        SELECT @partitionSQL = 'Select @partitionCount_OUT = Count(*)
                                    From ' + @databaseName + '.sys.partitions
                                    Where object_id = ' + CAST(@objectID AS VARCHAR(10)) + '
                                        And index_id = ' + CAST(@indexID AS VARCHAR(10)) + ';'
            , @partitionSQL_Param = '@partitionCount_OUT int OutPut';
 
        EXECUTE SP_EXECUTESQL @partitionSQL, @partitionSQL_Param, @partitionCount_OUT = @partitionCount OUTPUT;
 
        IF @debugMode = 1 RAISERROR('  Seeing if there''s any LOBs to be handled...', 0, 42) WITH NoWait;
 
        /* Determine if the table contains LOBs */
        SELECT @LOB_SQL = ' Select @containsLOB_OUT = Count(*)
                            From ' + @databaseName + '.sys.columns With (NoLock) 
                            Where [object_id] = ' + CAST(@objectID AS VARCHAR(10)) + '
                                And (system_type_id In (34, 35, 99)
                                        Or max_length = -1);'
                            /*  system_type_id --> 34 = image, 35 = text, 99 = ntext
                                max_length = -1 --> varbinary(max), varchar(max), nvarchar(max), xml */
                , @LOB_SQL_Param = '@containsLOB_OUT int OutPut';
 
        EXECUTE SP_EXECUTESQL @LOB_SQL, @LOB_SQL_Param, @containsLOB_OUT = @containsLOB OUTPUT;
 
        IF @debugMode = 1 RAISERROR('  Building our SQL statements...', 0, 42) WITH NoWait;
 
        /* If there's not a lot of fragmentation, or if we have a LOB, we should reorganize */
        IF @fragmentation < @rebuildThreshold Or @containsLOB >= 1 Or @partitionCount > 1
        BEGIN
 
            SET @sqlCommand = N'Alter Index ' + @indexName + N' On ' + @databaseName + N'.' 
                                + @schemaName + N'.' + @objectName + N' ReOrganize';
 
            /* If our index is partitioned, we should always reorganize */
            IF @partitionCount > 1
                SET @sqlCommand = @sqlCommand + N' Partition = ' 
                                + CAST(@partitionNumber AS NVARCHAR(10));
 
        END;
 
        /* If the index is heavily fragmented and doesn't contain any partitions or LOB's, rebuild it */
        IF @fragmentation >= @rebuildThreshold And IsNull(@containsLOB, 0) != 1 And @partitionCount <= 1
        BEGIN
 
            /* Set online rebuild options; requires Enterprise Edition */
            IF @onlineRebuild = 1 And @editionCheck = 1 
                SET @rebuildCommand = N' Rebuild With (Online = On';
            ELSE
                SET @rebuildCommand = N' Rebuild With (Online = Off';
 
            /* Set processor restriction options; requires Enterprise Edition */
            IF @maxDopRestriction IS Not Null And @editionCheck = 1
                SET @rebuildCommand = @rebuildCommand + N', MaxDop = ' + CAST(@maxDopRestriction AS VARCHAR(2)) + N')';
            ELSE
                SET @rebuildCommand = @rebuildCommand + N')';
 
            SET @sqlCommand = N'Alter Index ' + @indexName + N' On ' + @databaseName + N'.'
                            + @schemaName + N'.' + @objectName + @rebuildCommand;
 
        END;
 
        /* Are we executing the SQL?  If so, do it */
        IF @executeSQL = 1
        BEGIN
 
            IF @debugMode = 1 RAISERROR('  Executing SQL statements...', 0, 42) WITH NoWait;
 
            /* Grab the time for logging purposes */
            SET @dateTimeStart  = GETDATE();
 
            /* Log our actions */
            INSERT INTO dbo.dba_indexDefragLog
            (
                  databaseID
                , databaseName
                , objectID
                , objectName
                , indexID
                , indexName
                , partitionNumber
                , fragmentation
                , page_count
                , dateTimeStart
            )
            SELECT
                  @databaseID
                , @databaseName
                , @objectID
                , @objectName
                , @indexID
                , @indexName
                , @partitionNumber
                , @fragmentation
                , @pageCount
                , @dateTimeStart;
 
            SET @indexDefrag_id = SCOPE_IDENTITY();
 
            /* Execute our defrag! */
            EXECUTE SP_EXECUTESQL @sqlCommand;
            SET @dateTimeEnd  = GETDATE();
 
            /* Update our log with our completion time */
            UPDATE dbo.dba_indexDefragLog
            SET dateTimeEnd = @dateTimeEnd
                , durationSeconds = DATEDIFF(SECOND, @dateTimeStart, @dateTimeEnd)
            WHERE indexDefrag_id = @indexDefrag_id;
 
            /* Just a little breather for the server */
            WAITFOR Delay @defragDelay;
 
            /* Print if specified to do so */
            IF @printCommands = 1
                PRINT N'Executed: ' + @sqlCommand;
        END
        ELSE
        /* Looks like we're not executing, just printing the commands */
        BEGIN
            IF @debugMode = 1 RAISERROR('  Printing SQL statements...', 0, 42) WITH NoWait;
 
            IF @printCommands = 1 PRINT IsNull(@sqlCommand, 'error!');
        END
 
        IF @debugMode = 1 RAISERROR('  Updating our index defrag status...', 0, 42) WITH NoWait;
 
        /* Update our index defrag list so we know we've finished with that index */
        UPDATE #indexDefragList
        SET defragStatus = 1
        WHERE databaseID       = @databaseID
          And objectID         = @objectID
          And indexID          = @indexID
          And partitionNumber  = @partitionNumber;
 
    END
 
    /* Do we want to output our fragmentation results? */
    IF @printFragmentation = 1
    BEGIN
 
        IF @debugMode = 1 RAISERROR('  Displaying fragmentation results...', 0, 42) WITH NoWait;
 
        SELECT databaseID
            , databaseName
            , objectID
            , objectName
            , indexID
            , indexName
            , fragmentation
            , page_count
        FROM #indexDefragList;
 
    END;
 
    /* Do we want to rebuild stats? */
    IF @rebuildStats = 1
    BEGIN
 
        WHILE Exists(SELECT TOP 1 * FROM #databaseList WHERE statsStatus = 0)
        BEGIN
 
            /* Build our SQL statement to update stats */
            SELECT TOP 1 @rebuildStatsSQL = 'Use [' + databaseName + ']; ' + 
                                            'Execute sp_updatestats;'
                    , @rebuildStatsID = databaseID
            FROM #databaseList
            WHERE statsStatus = 0;
 
            SET @debugMessage = 'Rebuilding Statistics: ' + @rebuildStatsSQL;
 
            IF @debugMode = 1 RAISERROR(@debugMessage, 0, 42) WITH NoWait;
 
            /* Execute our stats update! */
            EXECUTE SP_EXECUTESQL @rebuildStatsSQL;
 
            /* Keep track of which databases have been updated */
            UPDATE #databaseList 
            SET statsStatus = 1
            WHERE databaseID = @rebuildStatsID;
 
        END;
    END;
 
    /* When everything is said and done, make sure to get rid of our temp table */
    DROP TABLE #indexDefragList;
    DROP TABLE #databaseList;
    DROP TABLE #processor;
 
    IF @debugMode = 1 RAISERROR('DONE!  Thank you for taking care of your indexes!  :)', 0, 42) WITH NoWait;
 
    SET NOCOUNT OFF;
    RETURN 0
END
Go
 
SET Quoted_Identifier OFF 
SET ANSI_Nulls ON
Go

Webcast Tomorrow!

Michelle Ufford — Wed, 27 May 2009 20:09:30 +0000

I'm excited to be doing a webcast tomorrow with the infamous illustrious Brent Ozar for Quest's Pain-of-the-Week. The title is "Getting Started with SQL Server Management Studio," and as you've probably gathered, it's pretty entry-level stuff. If you read my blog, then chances are you don't need to watch this webcast. But if you know anyone who's trying to learn SQL Server or is trying to make the upgrade from 2000 to 2005/2008, this may be a good webcast for them.

I've also got a few other speaking engagements coming up:

June 2nd: Cedar Valley .NET User Group
I'll be reprising my Iowa Code Camp presentation on "SQL Server for the .NET Developer" for CVINETA. This presentation focuses on what you need to know about good table design, indexing strategies, and fragmentation... you know, what you wish every .NET developer knew about SQL Server.

June 11th: PoTW: Time-Saving SQL Server Management Studio Tips & Tricks
I'll also be doing this webcast with @BrentO as a follow-up to our webcast tomorrow. It will focus on how to save time and improve your sanity by using some neat little tricks in SSMS 2008.

BOL 2008 Update Released

Michelle Ufford — Fri, 22 May 2009 21:39:22 +0000

If you haven't heard, Microsoft released an update to Books Online for SQL Server 2008 yesterday. You can find the download here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=765433f7-0983-4d7a-b628-0a98145bcb97&displaylang=en

Overhead in Non-Unique Clustered Indexes

Michelle Ufford — Thu, 21 May 2009 13:22:29 +0000

I've received a couple of questions regarding my article, Performance Considerations of Data Types, and the overhead associated with non-unique clustered indexes. I started to respond via e-mail, but my response was so long I decided to turn it into a blog post instead.

I should start by clarifying that non-unique clustered indexes do not necessarily consume more space and overhead; it depends on the data stored. If you have duplicate clustered key values, the first instance of the value will be handled as though it were unique. Any subsequent values, however, will incur overhead to manage the uniquifier that SQL Server adds to maintain row uniqueness. This same overhead is also incurred in non-clustered indexes, too, adding to the overall expense of this approach.

I think it helps to actually look at the data, so let's walk through a few different common scenarios. We'll create a table with a unique clustered index, a table with a non-unique clustered index but no duplicates, and a table with duplicate key values.

Also, a little warning that I started to write this in SQL Server 2008, and since I'm on a 2008 kick, I decided to leave it that way. You can modify this pretty easily to work in 2005, if necessary.

USE sandbox;
Go
 
/* Unique, clustered index, no duplicate values */
CREATE TABLE dbo.uniqueClustered
(
      myDate    DATE        Not Null
    , myNumber  INT         Not Null
    , myColumn  CHAR(995)   Not Null
);
 
CREATE UNIQUE CLUSTERED INDEX CIX_uniqueClustered
    ON dbo.uniqueClustered(myDate);
 
 
/* Non-unique clustered index, but no duplicate values */
CREATE TABLE dbo.nonUniqueNoDups
(
      myDate    DATE        Not Null
    , myNumber  INT         Not Null
    , myColumn  CHAR(995)   Not Null
);
 
CREATE CLUSTERED INDEX CIX_nonUniqueNoDups
    ON dbo.nonUniqueNoDups(myDate);
 
 
/* Non-unique clustered index, duplicate values */
CREATE TABLE dbo.nonUniqueDuplicates
(
      myDate    DATE        Not Null
    , myNumber  INT         Not Null
    , myColumn  CHAR(995)   Not Null
);
 
CREATE CLUSTERED INDEX CIX_nonUniqueDuplicates
    ON dbo.nonUniqueDuplicates(myDate);

I'm going to use the date data type in 2008 for my clustered index key. To ensure uniqueness for the first two tables, I'll iterate through a few years' worth of dates. This is typical of what you may see in a data mart, where you'd have one record with an aggregation of each day's data. For the table with duplicate values, I'm going to insert the same date for each row.

/* Populate some test data */
SET NOCOUNT ON;
DECLARE @myDate DATE = '1990-01-01'
    , @myNumber INT = 1;
 
WHILE @myDate < '2010-01-01'
BEGIN
 
    INSERT INTO dbo.uniqueClustered
    SELECT @myDate, @myNumber, 'data';
 
    INSERT INTO dbo.nonUniqueNoDups
    SELECT @myDate, @myNumber, 'data';
 
    INSERT INTO dbo.nonUniqueDuplicates
    SELECT '2009-01-01', @myNumber, 'data';
 
    SELECT @myDate = DATEADD(DAY, 1, @myDate)
        , @myNumber += 1;
 
END;

After running the above script, each table should have 7,305 records. This is obviously pretty small for a table, but it'll serve our purposes. Now let's take a look at the size of our tables:

/* Look at the details of our indexes */
 
/* Unique, clustered index, no duplicate values */
SELECT 'unique' AS 'type', page_count, avg_page_space_used_in_percent, record_count
    , min_record_size_in_bytes, max_record_size_in_bytes
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID(N'uniqueClustered'), Null, Null, N'Detailed') 
WHERE index_level = 0
UNION All
/* Non-unique clustered index, but no duplicate values */
SELECT 'non-unique, no dups', page_count, avg_page_space_used_in_percent, record_count
    , min_record_size_in_bytes, max_record_size_in_bytes
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID(N'nonUniqueNoDups'), Null, Null, N'Detailed') 
WHERE index_level = 0
UNION All
/* Non-unique clustered index, duplicate values */
SELECT 'duplicates', page_count, avg_page_space_used_in_percent, record_count
    , min_record_size_in_bytes, max_record_size_in_bytes
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID(N'nonUniqueDuplicates'), Null, Null, N'Detailed') 
WHERE index_level = 0;

Here's the results:

type                page_count           avg_page_space_used_in_percent record_count         min_record_size_in_bytes max_record_size_in_bytes
------------------- -------------------- ------------------------------ -------------------- ------------------------ ------------------------
unique              914                  99.8055102545095               7305                 1009                     1009
non-unique, no dups 914                  99.8055102545095               7305                 1009                     1009
duplicates          1044                 88.066036570299                7305                 1009                     1017

I want to point out a couple of things. First, there is no difference in the number of pages between the non-unique clustered index with no duplicates ([nonUniqueNoDups]) and the unique clustered index ([uniqueClustered]). The table with duplicate clustered key values, however, requires 14% more pages to store the same amount of data. Secondly, the [max_record_size_in_bytes] of the [nonUniqueDuplicates] table is 8 bytes more than that of the other two. We'll discuss why in a minute.

Now let's take a look at the actual data pages. For this, I'm going to use my page internals proc.

Execute dbo.dba_viewPageData_sp
      @databaseName = 'sandbox'
    , @tableName    = 'sandbox.dbo.uniqueClustered'
    , @indexName    = 'CIX_uniqueClustered';

I'm not going to post the entire results here, but I want to draw your attention to "m_slotCnt = 8", which is near the top of the page. That means 8 records are stored on this page. Also, when you look near the end of the first record (Slot 0), you should see the following results:

Slot 0 Offset 0x60 Length 1009
Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP     Record Size = 1009
Memory Dump @0x00A9C060
00000000:   1000ee03 c3150b01 00000064 61746120 †..î.Ã......data 
[...]
000003F0:   00†††††††††††††††††††††††††††††††††††.  
 
Slot 0 Column 1 Offset 0x4 Length 3 Length (physical) 3
myDate = 1990-01-01                  
 
Slot 0 Column 2 Offset 0x7 Length 4 Length (physical) 4
myNumber = 1                         
 
Slot 0 Column 3 Offset 0xb Length 995 Length (physical) 995
myColumn = data

Now let's look at the table that has a non-unique clustered index but no duplicates:

EXECUTE dbo.dba_viewPageData_sp
      @databaseName = 'sandbox'
    , @tableName    = 'sandbox.dbo.nonUniqueNoDups'
    , @indexName    = 'CIX_nonUniqueNoDups';

The m_slotCnt count is also 8 for this page. This time, let's glance at the first and second records (Slot 0 and 1 respectively):

Slot 0 Offset 0x60 Length 1009
Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP     Record Size = 1009
Memory Dump @0x62FDC060
00000000:   1000ee03 c3150b01 00000064 61746120 †..î.Ã......data    
[...]
000003F0:   00†††††††††††††††††††††††††††††††††††.                        
 
Slot 0 Column 0 Offset 0x0 Length 4 Length (physical) 0
UNIQUIFIER = 0                       
 
Slot 0 Column 1 Offset 0x4 Length 3 Length (physical) 3
myDate = 1990-01-01                  
 
Slot 0 Column 2 Offset 0x7 Length 4 Length (physical) 4
myNumber = 1                         
 
Slot 0 Column 3 Offset 0xb Length 995 Length (physical) 995
myColumn = data
 
 
Slot 1 Offset 0x451 Length 1009
Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP     Record Size = 1009
Memory Dump @0x62FDC451
 
00000000:   1000ee03 c4150b02 00000064 61746120 †..î.Ä......data          
[...]
000003F0:   00†††††††††††††††††††††††††††††††††††.                        
 
Slot 1 Column 0 Offset 0x0 Length 4 Length (physical) 0
UNIQUIFIER = 0                       
 
Slot 1 Column 1 Offset 0x4 Length 3 Length (physical) 3
myDate = 1990-01-02                  
 
Slot 1 Column 2 Offset 0x7 Length 4 Length (physical) 4
myNumber = 2                         
 
Slot 1 Column 3 Offset 0xb Length 995 Length (physical) 995
myColumn = data

We now see a new addition to the row, "UNIQUIFIER = 0." This is SQL Server's way of managing row uniqueness internally. You'll notice that, because the clustered key values are unique, the UNIQUIFIER is set to 0 and the row size is still 1009; for all intents and purposes, the UNIQUIFIER is not consuming any space.

Update: The DBCC God himself, Paul Randal, explained that non-dupes actually have a NULL UNIQUIFIER, which DBCC PAGE displays as a 0. Thanks for explaining, Paul! I wondered about that but chalked it up to SQL voodoo.

Now let's look at our final case, a non-unique clustered index with duplicate key values:

EXECUTE dbo.dba_viewPageData_sp
      @databaseName = 'sandbox'
    , @tableName    = 'sandbox.dbo.nonUniqueDuplicates'
    , @indexName    = 'CIX_nonUniqueDuplicates';

Here's where things get interesting. The m_slotCnt value is now 7, which means we're now storing 1 record less per page. Let's look at the details:

Slot 0 Offset 0x60 Length 1009
Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP     Record Size = 1009
Memory Dump @0x00A9C060
00000000:   1000ee03 df300b01 00000064 61746120 †..î.ß0.....data  
[...]
000003F0:   00†††††††††††††††††††††††††††††††††††.                        
 
Slot 0 Column 0 Offset 0x0 Length 4 Length (physical) 0
UNIQUIFIER = 0                       
 
Slot 0 Column 1 Offset 0x4 Length 3 Length (physical) 3
myDate = 2009-01-01                  
 
Slot 0 Column 2 Offset 0x7 Length 4 Length (physical) 4
myNumber = 1                         
 
Slot 0 Column 3 Offset 0xb Length 995 Length (physical) 995
myColumn = data
 
 
Slot 1 Offset 0x451 Length 1017
 
Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS
Record Size = 1017                   
Memory Dump @0x00A9C451
00000000:   3000ee03 df300b02 00000064 61746120 †0.î.ß0.....data  
[...]
000003F0:   000100f9 03010000 00†††††††††††††††††...ù.....                
 
Slot 1 Column 0 Offset 0x3f5 Length 4 Length (physical) 4
UNIQUIFIER = 1                       
 
Slot 1 Column 1 Offset 0x4 Length 3 Length (physical) 3
myDate = 2009-01-01                  
 
Slot 1 Column 2 Offset 0x7 Length 4 Length (physical) 4
myNumber = 2                         
 
Slot 1 Column 3 Offset 0xb Length 995 Length (physical) 995
myColumn = data

The first record, Slot 0, looks exactly the same as in the previous table; the UNIQUIFIER is 0 and the row size is 1009. The second record (Slot 1), however, now has a UNIQUIFIER value of 1 and the row size is 1017. If you notice, the "Record Attributes" of Slot 1 are also different, with the addition of "VARIABLE_COLUMNS." This is because the UNIQUIFIER is stored as a variable column. The extra 8 bytes of overhead break down to 4 bytes to store the UNIQUIFIER, 2 bytes to store the variable column offset, and 2 bytes to store the variable count. The tables we created used all fixed-length columns; you may notice some minor overhead differences if your table already contains variable columns.

To summarize, there is indeed a difference in the page structure between a unique clustered index and a non-unique clustered index; however, there's only a possible performance and space impact when storing duplicate clustered key values. So there you go, more detail than you ever wanted to know about clustered indexes and uniqueness!

Filtered Indexes: What You Need To Know

Michelle Ufford — Mon, 20 Apr 2009 17:32:56 +0000

Filtered indexes are probably my favorite feature in 2008. That's saying a lot, since there are so many great new features to choose from. In this post, I want to explore a little about how filtered indexes work, how they can be applied, and some of the "gotchas" to be aware of.

First, for those of you who may not yet know about filtered indexes, allow me enlighten you. In short, filtered indexes allow you to create an index on a subset of data using a filtering predicate. Filters can only be applied to non-clustered indexes. The general syntax of a filtered index is:

CREATE NONCLUSTERED INDEX [index_name]
ON [table_name] ([column_list])
Include ([column_list])
WHERE [filtered_criteria];

For our purposes, we're going to be working with the Sales.SalesOrderDetail table in the AdventureWorks database. Let's look at a specific example. Suppose we have a query that regularly searches on the [SpecialOfferID] column.

SELECT SalesOrderID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID;

We notice that there's no covering index for this query by looking at the actual execution plan:

Query Plan - Clustered Scan

If this is a commonly executed query, then we'd probably want to toss an index on it. Before we get started, let's take a look at what the distribution of values are on that column:

SELECT SpecialOfferID
    , COUNT(*) AS 'rows'
FROM Sales.SalesOrderDetail
GROUP BY SpecialOfferID
ORDER BY COUNT(*) DESC;

Our distribution of values is:

SpecialOfferID rows
-------------- -----------
1              115884
2              3428
3              606
13             524
14             244
16             169
7              137
8              98
11             84
4              80
9              61
5              2

As you can see, [SpecialOfferID] = 1 accounts for 96% of our values. In 2005, we'd create an index that may look something like this:

CREATE NONCLUSTERED INDEX IX_Sales_SalesOrderDetail_SpecialOfferID
    ON Sales.SalesOrderDetail(SpecialOfferID)
    Include (SalesOrderID, LineTotal);

Now if we re-run our original query, this is what we see:

Indexed Query Plan

So we're now performing a non-clustered index seek instead of a clustered index scan. Already this results in some pretty significant performance improvements. To see this, we're going to use the INDEX query hint to force an index scan. We're also going to use the DBCC command DROPCLEANBUFFERS, which will allow us to clear the buffer cache and better examine what's happening with our IO.

SET STATISTICS IO ON;
 
DBCC DropCleanBuffers;
 
SELECT SalesOrderID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail WITH 
    (INDEX(PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID))
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID;
 
DBCC DropCleanBuffers;
 
SELECT SalesOrderID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID;
 
SET STATISTICS IO OFF;

Clustered Index Scan:
Table 'SalesOrderDetail'. Scan count 1, logical reads 1240, physical reads 17, read-ahead reads 1242...
 
NonClustered Index Seek:
Table 'SalesOrderDetail'. Scan count 2, logical reads 30, physical reads 4, read-ahead reads 480...

As you can see, the non-clustered (NC) index seek performs quite a bit better. Now let's create a filtered index and explore what happens:

CREATE NONCLUSTERED INDEX FIX_Sales_SalesOrderDetail_SpecialOfferID_Filtered
    ON Sales.SalesOrderDetail(SalesOrderID)
    Include (LineTotal)
    WHERE SpecialOfferID <> 1;

First, let's look at the pages consumed by each index:

SELECT i.name, ddips.index_depth, ddips.index_level
    , ddips.page_count, ddips.record_count
FROM sys.indexes AS i
Join sys.dm_db_index_physical_stats(DB_ID(), 
    OBJECT_ID(N'Sales.SalesOrderDetail'), Null, Null, N'Detailed') AS ddips
    ON i.OBJECT_ID = ddips.OBJECT_ID
    And i.index_id = ddips.index_id
WHERE i.name In ('IX_Sales_SalesOrderDetail_SpecialOfferID'
    , 'FIX_Sales_SalesOrderDetail_SpecialOfferID_Filtered'
    , 'PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID')
    AND ddips.index_level = 0;

name                                                       index_depth index_level page_count  record_count
---------------------------------------------------------- ----------- ----------- ----------- --------------------
PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID        3           0           1234        121317
IX_Sales_SalesOrderDetail_SpecialOfferID                   3           0           480         121317
FIX_Sales_SalesOrderDetail_SpecialOfferID_Filtered         2           0           19          5433

If you scroll over, you'll see that the clustered index consumes the most pages, naturally. The non-filtered NC index consumes less pages than the clustered index because it's narrower; however, it still consumes more pages than the filtered index because it's storing every data row. The filtered index, with only 5433 rows stored, is by far our smallest index, consuming 96% less space than our non-filtered NC index.

Because we're using less space to store this index, we should also see an equivalent performance boost. Let's verify that this is the case:

SET STATISTICS IO ON;
 
DBCC DropCleanBuffers;
 
SELECT SalesOrderID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail WITH (INDEX(IX_Sales_SalesOrderDetail_SpecialOfferID))
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID;
 
DBCC DropCleanBuffers;
 
SELECT SalesOrderID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID;
 
SET STATISTICS IO OFF;

NonClustered Index Seek:
Table 'SalesOrderDetail'. Scan count 2, logical reads 30, physical reads 4, read-ahead reads 480
 
Filtered Index Scan:
Table 'SalesOrderDetail'. Scan count 1, logical reads 24, physical reads 2, read-ahead reads 22

Filtered Query Plan

As expected, we get the best results with our filtered index scan.

You'll notice that I did *not* create the index on the [SpecialOfferID] column like I did in [IX_Sales_SalesOrderDetail_SpecialOfferID]. This is because my query doesn't care what my [SpecialOfferID] value is, just as long as it's not equal to 1. My non-filtered NC index was created on [SpecialOfferID] because it needed to navigate the B-TREE to find the records where [SpecialOfferID] <> 1. With my filtered index, the query optimizer knows that all of my records already meet the criteria, so doesn't need to navigate through the index to find the matching results.

We could choose to include the [SpecialOfferID] data in our filtered index, but we'd most likely want to make it an included column rather than part of the index key. In fact, it's important to note that, if I don't add [SpecialOfferID] as an included column and I want to return it in the results, i.e.

SELECT SalesOrderID
    , SpecialOfferID
    , COUNT(*) AS 'CountOfLineItem'
    , SUM(LineTotal) AS 'SumOfLineTotal'
FROM Sales.SalesOrderDetail
WHERE SpecialOfferID <> 1
GROUP BY SalesOrderID
    , SpecialOfferID;

my filtered index will not be used and I will instead scan on the clustered index once more (assuming [IX_Sales_SalesOrderDetail_SpecialOfferID] does not exist). This is because the filtering criteria is not included anywhere on the actual index page. This is actually good news, in my opinion, since it allows you to create even leaner indexes. And like I already mentioned, if you do need the data returned, you can always add the filtering criteria as included columns.

What if you're trying to find out whether or not an index is filtered, and what it's filtered on? The sys.indexes catalog view has been updated in 2008 to include this information:

SELECT name, has_filter, filter_definition
FROM sys.indexes 
WHERE name In ('IX_Sales_SalesOrderDetail_SpecialOfferID'
    , 'FIX_Sales_SalesOrderDetail_SpecialOfferID_Filtered'
    , 'PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID');

name                                                   has_filter filter_definition
------------------------------------------------------ ---------- -------------------------
FIX_Sales_SalesOrderDetail_SpecialOfferID_Filtered     1          ([SpecialOfferID]<>(1))
IX_Sales_SalesOrderDetail_SpecialOfferID               0          NULL
PK_SalesOrderDetail_SalesOrderID_SalesOrderDetailID    0          NULL

I personally recommend Kimberly Tripp's system stored proc, sp_helpindex2. It returns a lot of good information about your indexes, such as included columns and filtering criteria.

That's all I have for today. Hopefully, you now understand how powerful filtered indexes can be. When used properly, filtered indexes can use less space, consume less IO, and improve overall query performance.