Category: SQL

Indexes, SQL

How to identify redundant/duplicate indexes in SQL Server

What are redundant indexes in SQL Server?

Identifying redundant indexes is a very important task for a DBA. I personally do not find any reason of having redundant indexes on databases. Otherwise, it is an over head to the system like maintaining the index/storage space etc.

The challenge is to identify a redundant/duplicate index in SQL Server? If we do not understand the meaning of redundant/duplicate, then we might end up with removing important indexes and it can lead to performance degradation.

I would like to list out few considerations to categorize indexes as redundant; if:

1. An index has Same key columns in the same order with another index

2. An index has Key columns those are left based subset of another index

3. Meeting all of the above, ONLY for similar index types.(Clustered and no clustered are not to be considered as duplicate indexes)

4. Meeting all of the above, And the Key Columns specified with same ordering (ASC/DESC)

The below query does not identify the duplicate index, but it gives you enough information with which we can easily identify the duplicate indexes in your system.

Note:(String_AGG is new function in SQL Server 2017) 

Select Object_Schema_name(ix.object_id) Schema_Name,
		Object_name(ix.object_id) Object_Name,ix.name,ix.type_desc ,
		string_agg(Cast(c.name as nvarchar(MAX)) + ' (' + case when is_descending_key = 0  then 'ASC' Else 'DESC' END + ') ',',') 
		within group(Order by ixc.key_ordinal asc) As KeyCols
From
	sys.indexes ix
	inner join sys.index_columns ixc on ix.index_id = ixc.index_id and ix.object_id = ixc.object_id
	inner join sys.columns c on ixc.object_id = c.object_id and c.column_id = ixc.column_id 
Where objectpropertyex(ix.object_id,'IsMSShipped') =0 
		And ixc.is_included_column=0 and ix.index_id <> 1 
		Group by ix.object_id,ix.name,ix.type_desc
Order by 1 asc,2 asc

I’d like to grow my readership. If you enjoyed this blog post, please share it with your friends!

Errors and Exceptions, SQL, SQL Job

The enlist operation failed (reason: SQLServerAgent Error: The target server cannot establish an encrypted connection to the master server ‘Servername’. Make sure that the MsxEncryptChannelOptions registry subkey is set correctly on the target server.) (Microsoft SQL Server, Error: 22026)

While configuring SQL Server Multi Server Administration: Master and Target Servers in SQL Agent, we encountered an error as below:

TITLE: Microsoft.SqlServer.Smo
——————————

MSX enlist failed for JobServer ‘TargetServer’.

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=14.0.17289.0+((SSMS_Rel_17_4).181117-0805)&
EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=MSX+enlist+JobServer&LinkId=20476

——————————
ADDITIONAL INFORMATION:

An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)

——————————

The enlist operation failed
(reason: SQLServerAgent Error: The target server cannot establish an encrypted connection to the master server ‘MasterServer’.
Make sure that the MsxEncryptChannelOptions registry subkey is set correctly on the target server.) (Microsoft SQL Server, Error: 22026)

Root cause Analysis & Resolution

As per the investigation, it is identified as an issue associated with a registry value on the target server MsxEncryptChannelOptions. When we configure the set up, it tries to establish the connection between master and targets in a secure channel with full SSL encryption. And if SSL encryption is not enabled between servers/instances, then this setting has to be changed in target servers using the registry. You can change the registry value here: \HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\InstanceName\SQLServerAgent. Please note the default value is 2. We need to change it to 0, however, this change needs to be verified and evaluated with you security norms and standards.

Refer Also:

Set Encryption Options on Target Servers

SQL, Table Partitioning

Stairway to SQL Server Table Partitioning – How do we partition an existing table

This post explains how to implement table partitioning for an existing table in SQL Server. Let us create a table as below for our explanation.

Create Table and Populate table data

Create Table AuditData (ID int identity(1,1) Primary Key not null, AuditDate Datetime not null)

Insert into AuditData Values(getdate() - (365*5)) ,(getdate() - (365*4)) ,(getdate() - (365*3))
,(getdate() - (365*2)) ,(getdate() - (365*1)) ,(getdate()) ,(getdate() + (365*5))
,(getdate() + (365*4)) ,(getdate() + (365*3)) ,(getdate() + (365*2)) ,(getdate() + (365*1))

--Check the data
Select * From AuditData

The above code will create a table called AuditData and inserts a single row for every previous and upcoming 5 years from now. Please note this test data is created on year 2020, so there might be difference in the data at the time you refer this blog.

Check the partition and its data allocation

select Object_name(p.object_id) Table_Name ,(Select name From sys.indexes where Object_id = p.object_id and index_id = p.index_id) 'Index_name',
partition_number, lv.value leftValue, rv.value rightValue,p.rows,
s.name 'Partition_Schema_Name',f.name 'Partition_Function_Name'
from sys.partitions p
join sys.allocation_units a on p.hobt_id = a.container_id
join sys.indexes i on p.object_id = i.object_id
Left join sys.partition_schemes s on i.data_space_id = s.data_space_id
left join sys.partition_functions f on s.function_id = f.function_id
left join sys.partition_range_values rv on f.function_id = rv.function_id and p.partition_number = rv.boundary_id
left join sys.partition_range_values lv on f.function_id = lv.function_id and p.partition_number - 1 = lv.boundary_id
where p.object_id = object_id('AuditData')

From the above, its clear that the index created on the table as part of Primary key has only one partition and all the data is part of that partition. Now, let us quickly convert this existing table as as partitioned table. To do so, we need to create partition function and partition schema as below. The function fn_AuditDate has been created on Datetime field so that in our example, we can create partition based on AuditDate. 
--Create partition function on Datetime
Create partition function fn_AuditDate (Datetime) as
Range right for values('20150101', '20160101','20170101','20180101', '20190101','20200101')
/*Please note this test data is created on year 2020, so there might be difference in the data at the time you refer this blog.*/

--Create partition Schema to associate the function
Create partition scheme sc_AuditDate As 
Partition  fn_AuditDate ALL to ([Primary])
On successful execution, we will get the below Message:
Partition scheme ‘sc_AuditDate’ has been created successfully. ‘PRIMARY’ is marked as the next used filegroup in partition scheme ‘sc_AuditDate’.

Recreate clustered index to make the partition column as part of clustered index

ALTER TABLE dbo.AuditData DROP CONSTRAINT [PK__AuditDat__3214EC2760DC1A18]
GO
ALTER TABLE dbo.AuditData ADD CONSTRAINT [PK__AuditDat__3214EC2760DC1A18] PRIMARY KEY NONCLUSTERED  (ID)
   ON [PRIMARY]
GO
CREATE CLUSTERED INDEX IX_AuditData_AuditDate_ID ON dbo.AuditData (AuditDate)
  ON sc_AuditDate(AuditDate)
GO

Now, it is important to make sure that the clustered index to be recreated on partition schema to partition the AuditData table. If you look at the above example, since the clustered index is created as part of Primary key creation, we need to drop the primary key and re create the primary key as non clustered index and create a separate clustered index on AuditDate. Note that the clustered index is created on partition that we created recently – sc_AuditDate.

Check the partition and its data allocation

select Object_name(p.object_id) 'Table_Name',(Select name From sys.indexes where Object_id = p.object_id and index_id = p.index_id) 'Index_name',
partition_number, lv.value leftValue, rv.value rightValue,p.rows,
s.name 'Partition_Schema_Name',f.name 'Partition_Function_Name'
from sys.partitions p
join sys.allocation_units a on p.hobt_id = a.container_id
join sys.indexes i on p.object_id = i.object_id
join sys.partition_schemes s on i.data_space_id = s.data_space_id
join sys.partition_functions f on s.function_id = f.function_id
left join sys.partition_range_values rv on f.function_id = rv.function_id and p.partition_number = rv.boundary_id
left join sys.partition_range_values lv on f.function_id = lv.function_id and p.partition_number - 1 = lv.boundary_id
where p.object_id = object_id('AuditData')

Now, you can see the clustered index has 7 partitions as per the range that we defined in the schema and function. The last partition contains all the data right to the range (eg: 2020/2021/2022/2023/2024/2025); 6 rows in our AuditData table.

Hope this explains how to partition an existing table simply. But wait, that may not be so easy as I explained for your real time scenario. You may want to implement partition on a table that has lots of data/ multiple indexes/foreign key relations defined etc. We need to carefully evaluate the steps to reduce the down time during the implementation of partition. I would suggest partitioning a table (especially bigger ones) should be an offline activity rather than an online for better performance and easy implementation and testing.

Cleanup Objects

Clean up is very important, so my test objects. 

DROP TABLE AuditData
DROP PARTITION SCHEME sc_AuditDate
DROP PARTITION FUNCTION fn_AuditDate

I’d like to grow my readership. If you enjoyed this blog post, please share it with your friends!

SCRIPT GALLERY, SQL

Search a value in your database in SQL Server
Here is a script to identify a value in your database. The script will identify the presence of the search value from all tables and generate select queries from the respective tables which you can execute and confirm. 

	DECLARE @STRINGTOLOOKFOR VARCHAR(500)
		,@TABLENAME SYSNAME
		,@FULLTABLENAME SYSNAME
        --<-- Give the value to search------->
	SELECT @STRINGTOLOOKFOR = '7F0B0DD5-22BC-421B-9416-3A7C24146A98'  
        --<-- Give the value to search------->

	DECLARE @COLUMNNAME NVARCHAR(128),
		@DATETYPE NVARCHAR(128),
		@ROW SMALLINT,
		@ROWCOUNT INT,
		@SQL NVARCHAR(1000)
		
	DECLARE STRING_FIND_CURSOR CURSOR FAST_FORWARD FOR 
		
	SELECT TABLE_NAME, TABLE_SCHEMA+'.'+TABLE_NAME FROM INFORMATION_SCHEMA.TABLES 
	WHERE TABLE_TYPE ='BASE TABLE' 
	
	OPEN STRING_FIND_CURSOR
	
	FETCH NEXT FROM STRING_FIND_CURSOR 
	INTO @TABLENAME, @FULLTABLENAME
	
	SET @STRINGTOLOOKFOR = @STRINGTOLOOKFOR 
	
	WHILE @@FETCH_STATUS = 0
	BEGIN
		SET @ROW = 1
	
		SELECT @ROWCOUNT = MAX([ORDINAL_POSITION])
		FROM [INFORMATION_SCHEMA].[COLUMNS]
		WHERE [TABLE_NAME] = @TABLENAME 
		GROUP BY [ORDINAL_POSITION]
	
		WHILE @ROW  <= @ROWCOUNT
                BEGIN 
		SELECT @COLUMNNAME =  QUOTENAME(COLUMN_NAME) ,
			@DATETYPE = [DATA_TYPE]
		FROM [INFORMATION_SCHEMA].[COLUMNS]
		WHERE [TABLE_NAME] = @TABLENAME 
			AND [ORDINAL_POSITION] = @ROW
		ORDER BY [ORDINAL_POSITION]
	
		SET @ROW = @ROW + 1  
			
		SET @SQL = NULL
	
		IF @DATETYPE IN ( N'CHAR', N'VARCHAR', N'TEXT')
			SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE PATINDEX(''%' + @STRINGTOLOOKFOR + '%'', ' + @COLUMNNAME + ') > 0'
			IF @DATETYPE IN (N'UNIQUEIDENTIFIER') AND @STRINGTOLOOKFOR LIKE '[A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9]'
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE' + @COLUMNNAME + ' = ''' + @STRINGTOLOOKFOR + ''''
			IF @DATETYPE IN (N'NCHAR', N'NVARCHAR', N'NTEXT')
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE PATINDEX(''%' + @STRINGTOLOOKFOR + '%'', CAST(' + @COLUMNNAME + ' AS TEXT)) > 0'
			
			IF @DATETYPE IN (N'SQL_VARIANT',N'SMALLINT',N'INT',N'BIGINT',N'TINYINT')
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE CONVERT(VARCHAR(8000),' + @COLUMNNAME + ') LIKE ''%'+ @STRINGTOLOOKFOR + '%'''
			
			IF @SQL IS NOT NULL
			BEGIN
				SET @SQL = 'IF EXISTS(' + @SQL + ') PRINT ''SELECT '+@COLUMNNAME+' FROM ' + @FULLTABLENAME + ' WHERE ' + @COLUMNNAME + ' LIKE ''''%'+@STRINGTOLOOKFOR+'%'''''''
				EXEC (@SQL)
			END
		END
	
		FETCH NEXT FROM STRING_FIND_CURSOR 
		INTO @TABLENAME, @FULLTABLENAME

	END
	
	CLOSE STRING_FIND_CURSOR
	DEALLOCATE STRING_FIND_CURSOR