Category: SCRIPT GALLERY

Understanding PIVOT operator in SQL Server with Examples

Today, we will quickly see PIVOT operator in SQL Server and why do we need this and its various forms of usages. To understand better, let us think of a transformation of data as below and see how do we approach further.

First off, let us create a CustomerPurchaseTable as below. Please note if you are not heard of DROP IF EXISTS yet, please refer here.

Drop Table if exists  CustomerPurchaseTable

create Table CustomerPurchaseTable( CustomerID int, FruitName varchar(100), Quantity int)
Insert into CustomerPurchaseTable Values
(100,'Apple',10),(100,'Grapes', 5),(101,'Orange',10),(102,'Pineapple',5),(102,'Grapes',10)

Select * From CustomerPurchaseTable

I remember when I started my career I was using SQL Server 7.0 and SQL 2000. It was not so easy task to achieve transformations as of today, particularly, the pivoting had achieved using Select ..CASE statements as below.

Select CustomerID, Sum(Case When FruitName = 'Apple' Then Quantity Else NULL End )'Apple',
				   Sum(Case when FruitName = 'Grapes' Then Quantity Else NULL End ) 'Grapes',
				   Sum(Case when FruitName = 'Orange' Then Quantity Else NULL End ) 'Orange'
From CustomerPurchaseTable
Group by CustomerID

Post SQL Server 2005 version release, there was a new operator introduced “PIVOT” which does the work with ease. I would say, its a fancy form of old way.But, at the same time, I would agree that its a saver at times, we would see that later part of this blog post.

Here is the syntax of PIVOT:

Syntax:
SELECT first_column AS ,
[pivot_value1], [pivot_value2], ... [pivot_value_n]
FROM 
() AS 
PIVOT 
(
 aggregate_function()
 FOR 
 IN ([pivot_value1], [pivot_value2], ... [pivot_value_n])
) AS ;

Select * From CustomerPurchaseTable
PIVOT (Sum(Quantity) For FruitName in ([Apple],[Grapes],[Orange]))as PivotTable

Now, let us look at a case how it would work for a non-fixed type of fruits in the above example? Then, this should be a dynamic transformation that might required. With old approach(prior to SQL Server 2005), this is a complicated one and hard to achieve with Dynamic Cross-Tabs/Pivot Tables. With the introduction of PIVOT that becomes handy. That does not mean that we do not need dynamic query, but it extremely handy to build dynamic list of columns and assign the value to the dynamic Query to produce the transformation. The generation of column list with quotename is much easier than generating the entire sql construct of CASE…WHEN as in the old way.

DECLARE 
    @columns NVARCHAR(MAX) = '', @sql     NVARCHAR(MAX) = '';

SELECT @columns += QUOTENAME(FruitName) + ','
FROM (Select Distinct FruitName From CustomerPurchaseTable) A
ORDER BY FruitName ASc

SET @columns = LEFT(@columns, LEN(@columns) - 1);
PRINT @columns;

Set @SQL = 'Select * From CustomerPurchaseTable
PIVOT (Sum(Quantity) For FruitName in (' + @columns + '))as PivotTable'

Exec sp_executeSQL @SQL

I’d like to grow my readership. If you enjoyed this blog post, please share it with your friends!

Search a value in your database in SQL Server

Here is a script to identify a value in your database. The script will identify the presence of the search value from all tables and generate select queries from the respective tables which you can execute and confirm.

	DECLARE @STRINGTOLOOKFOR VARCHAR(500)
		,@TABLENAME SYSNAME
		,@FULLTABLENAME SYSNAME
        --<-- Give the value to search------->
	SELECT @STRINGTOLOOKFOR = '7F0B0DD5-22BC-421B-9416-3A7C24146A98'  
        --<-- Give the value to search------->

	DECLARE @COLUMNNAME NVARCHAR(128),
		@DATETYPE NVARCHAR(128),
		@ROW SMALLINT,
		@ROWCOUNT INT,
		@SQL NVARCHAR(1000)
		
	DECLARE STRING_FIND_CURSOR CURSOR FAST_FORWARD FOR 
		
	SELECT TABLE_NAME, TABLE_SCHEMA+'.'+TABLE_NAME FROM INFORMATION_SCHEMA.TABLES 
	WHERE TABLE_TYPE ='BASE TABLE' 
	
	OPEN STRING_FIND_CURSOR
	
	FETCH NEXT FROM STRING_FIND_CURSOR 
	INTO @TABLENAME, @FULLTABLENAME
	
	SET @STRINGTOLOOKFOR = @STRINGTOLOOKFOR 
	
	WHILE @@FETCH_STATUS = 0
	BEGIN
		SET @ROW = 1
	
		SELECT @ROWCOUNT = MAX([ORDINAL_POSITION])
		FROM [INFORMATION_SCHEMA].[COLUMNS]
		WHERE [TABLE_NAME] = @TABLENAME 
		GROUP BY [ORDINAL_POSITION]
	
		WHILE @ROW  <= @ROWCOUNT
                BEGIN 
		SELECT @COLUMNNAME =  QUOTENAME(COLUMN_NAME) ,
			@DATETYPE = [DATA_TYPE]
		FROM [INFORMATION_SCHEMA].[COLUMNS]
		WHERE [TABLE_NAME] = @TABLENAME 
			AND [ORDINAL_POSITION] = @ROW
		ORDER BY [ORDINAL_POSITION]
	
		SET @ROW = @ROW + 1  
			
		SET @SQL = NULL
	
		IF @DATETYPE IN ( N'CHAR', N'VARCHAR', N'TEXT')
			SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE PATINDEX(''%' + @STRINGTOLOOKFOR + '%'', ' + @COLUMNNAME + ') > 0'
			IF @DATETYPE IN (N'UNIQUEIDENTIFIER') AND @STRINGTOLOOKFOR LIKE '[A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][-][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9][A-F,0-9]'
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE' + @COLUMNNAME + ' = ''' + @STRINGTOLOOKFOR + ''''
			IF @DATETYPE IN (N'NCHAR', N'NVARCHAR', N'NTEXT')
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE PATINDEX(''%' + @STRINGTOLOOKFOR + '%'', CAST(' + @COLUMNNAME + ' AS TEXT)) > 0'
			
			IF @DATETYPE IN (N'SQL_VARIANT',N'SMALLINT',N'INT',N'BIGINT',N'TINYINT')
				SET @SQL = 'SELECT * FROM ' + @FULLTABLENAME + ' WHERE CONVERT(VARCHAR(8000),' + @COLUMNNAME + ') LIKE ''%'+ @STRINGTOLOOKFOR + '%'''
			
			IF @SQL IS NOT NULL
			BEGIN
				SET @SQL = 'IF EXISTS(' + @SQL + ') PRINT ''SELECT '+@COLUMNNAME+' FROM ' + @FULLTABLENAME + ' WHERE ' + @COLUMNNAME + ' LIKE ''''%'+@STRINGTOLOOKFOR+'%'''''''
				EXEC (@SQL)
			END
		END
	
		FETCH NEXT FROM STRING_FIND_CURSOR 
		INTO @TABLENAME, @FULLTABLENAME

	END
	
	CLOSE STRING_FIND_CURSOR
	DEALLOCATE STRING_FIND_CURSOR
	
	

How to build comma separated string in SQL Server

Building a comma separated string is a very frequent&common requirement for SQL Server developers. Developers find different ways to achieve this like CLR/XML PATH/Co-related queries etc. With this post, we are going to see two options that are widely used XML PATH & string_AGG and a quick look at its performance comparison.

— Using STUFF & XML PATH

This is the most widely used method to build the comma separated string. In my personal experience, I observed many performance issues related to this method for a large set of data.


Drop Table if exists BuildCommaSeparatedString
Create Table BuildCommaSeparatedString(SessionID int, CourseID varchar(100))

Insert into BuildCommaSeparatedString Values 
(1,'English'),(1,'Maths'),(1,'Accountancy'),(2,'History'),(2,'Biology')

SELECT  SessionID,STUFF((SELECT  ',' + CourseID FROM BuildCommaSeparatedString EE
            WHERE  EE.SessionID=E.SessionID
            ORDER BY CourseID -- To order the Courses in order 
        FOR XML PATH('')), 1, 1, '') AS 'CommaseparatedString'
FROM BuildCommaSeparatedString E
GROUP BY E.SessionID


Sample Execution results — Using string_AGG in SQL Server 2017

With SQL Server 2017 release, there is a better way to do this using string_agg built-in function. The STRING_AGG() is an aggregate function that concatenates rows of strings into a single string, separated by a specified separator.

STRING_AGG ( input_string, separator ) [ order_clause ]

input_string - is the column name of any type that can convert to a comma separated string
separator - is the separator caharacter like , or ; etc
[ order_clause ] - specifies the sort order of concatenated results using WITHIN GROUP clause

WITHIN GROUP ( ORDER BY expression [ ASC | DESC ] )
Please note that, The STRING_AGG() ignores NULL and it does not add the separator for NULL when performing concatenation.

Drop Table if exists BuildCommaSeparatedString
Create Table BuildCommaSeparatedString(SessionID int, CourseID varchar(100))

Insert into BuildCommaSeparatedString Values 
(1,'English'),(1,'Maths'),(1,'Accountancy'),(2,'History'),(2,'Biology')

Select SessionID, STRING_AGG(CourseID,',') From BuildCommaSeparatedString Group by SessionID

--To build the comma separated string in an order way
Select SessionID, STRING_AGG(CourseID,',') within group (Order by CourseID asc) 
From BuildCommaSeparatedString Group by SessionID

Sample Execution results A quick look on Performance benefit

The below snapshot clearly indicates that the performance benefit of string_AGG built-in function over XML PATH approach.

Conclusion

If you are in SQL Server 2017 – use built-in function string_AGG instead of any other method. I have seen this outperform many times than a custom built code. If you do not agree or have any different experience, please share the specific case with an example for the benefits of readers. Thanks in advance!

How to remove special characters like TAB, Carriage Return, and Line Feed characters from string in SQL Server

Today, we are going to see a scenario where we need to remove special characters from a string in SQL Server. Let us discuss on few methods to achieve this “extended” trim functionality.

1. Option: Using REPLACE

REPLACE function can be effectively used to replace the special characters.

Script:

DECLARE @BaddataString NVARCHAR(max);
SET @BaddataString = 'my '+
			CHAR(10)+CHAR(10)+CHAR(10) +
			 'text   ' + 
			CHAR(9)+CHAR(10)+CHAR(13)+CHAR(32) + 
			 '     ' +
			CHAR(10)+CHAR(10)+CHAR(10);

SELECT 
           @BaddataString,
           LEN(@BaddataString)
           , REPLACE(REPLACE(REPLACE(RTRIM(@BaddataString),CHAR(9), ''),CHAR(10),''),CHAR(13),'') ,
           LEN(REPLACE(REPLACE(REPLACE(RTRIM(@BaddataString),CHAR(9), ''),CHAR(10),''),CHAR(13),''))


Here is the output of the below code:

2. Option: Using CDATA Another effective option to use CDATA method.

Script


/*
The below function replaces special characters like invisible TAB, Carriage Return, and Line Feed characters.
*/
CREATE FUNCTION dbo.FN_RemoveBadCharacter(@input VARCHAR(MAX))
   RETURNS VARCHAR(MAX)
AS
BEGIN 
   RETURN (SELECT CAST('' AS XML).value('(/r/text())[1] cast as xs:token?','VARCHAR(MAX)'));
END
GO
DECLARE @BaddataString NVARCHAR(max);
SET @BaddataString = 'my '+
						CHAR(10)+CHAR(10)+CHAR(10) +
					 'text   ' + 
						CHAR(9)+CHAR(10)+CHAR(13)+CHAR(32) + 
					 '     ' +
						CHAR(10)+CHAR(10)+CHAR(10);

SELECT		@BaddataString, 
		LEN(@BaddataString)
			, dbo.FN_RemoveBadCharacter(@BaddataString),
		LEN(dbo.FN_RemoveBadCharacter(@BaddataString))

Here is the output of the below code:

3. Option: Using CLR to define a function to remove

There are certain things its best to do at application code rather than T-SQL. SQL Server may not be the best place to formatting string, date etc. There will be lots of available resources just away from a google search.

Hope, you enjoy this post, please share your thoughts as always!

Generate Random characters in SQL Server

Here is a script to generate randon characters in SQL Server

It also allows you to generate the characters in random as well as required format. There are four types of format the function supports:

Formats:
1. ‘Proper’ – proper name form (i.e. Xxxxx)
2. ‘Upper’ – all uppercase (i.e. XXXXX)
3. ‘Lower’ – all lowercase (i.e. xxxxx)
5. ‘Mixed’ – randomly mixed case (i.e. xXxxxXXxx)


Create View dbo.Q_Random as 
	Select Rand() as [RandomNumber]
GO
Create Function dbo.fn_GenerateText 
				(@Length integer, @Format varchar(6) = 'Mixed')
Returns varchar(256)
as 
Begin

--	Formats:	
--		'Proper' - proper name form (i.e. Xxxxx)
--		'Upper'  - all uppercase (i.e. XXXXX)
--		'Lower'  - all lowercase (i.e. xxxxx)
--		'Mixed'  - randomly mixed case (i.e. xXxxxXXxx)
--		 null    - randomly mixed case (i.e. xXxXxxxxxX)
--

Declare 
	@RandomValue             varchar(256), 
	@Count                   integer,
	@RandomNumber            float, 
	@RandomNumberInteger     integer, 
	@CurrentCharacter        char(1),
	@ValidCharactersLength   integer,
	@ValidCharacters         varchar(255) 

Set @RandomValue = '';

If (@Length = 0) 
	Goto ReturnData 


If (@Format = 'Mixed') 
	Set @ValidCharacters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'; 
else
	Set @ValidCharacters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';


Set @ValidCharactersLength = Len(@ValidCharacters); 
Set @CurrentCharacter      = ''; 
Set @RandomNumber          = 0; 
Set @RandomNumberInteger   = 0; 

Set @Count = 1; 


While @Count <= @Length 
Begin 
	Set @RandomNumber = (Select RandomNumber from Q_Random); 

	Set @RandomNumberInteger = Convert(integer, ((@ValidCharactersLength - 1) * @RandomNumber + 1)); 
 
	Set @CurrentCharacter = SubString(@ValidCharacters, @RandomNumberInteger, 1); 

	Set @RandomValue = @RandomValue + @CurrentCharacter; 

	Set @Count = @Count + 1; 
End 


If @Format = 'Lower' 
	Set @RandomValue = Lower(@RandomValue); 

If @Format = 'Upper' 
	Set @RandomValue = Upper(@RandomValue); 

If @Format = 'Proper' 
	Set @RandomValue = Upper(Left(@RandomValue, 1)) + Substring(Lower(@RandomValue), 2, (@Length - 1));  

--	... or the default  gives random `casing`, and 'Mixed' gives random alphanumeric `casing`


ReturnData:

Return @RandomValue 

End 

Sample Executions:

Select dbo.fn_GenerateText(10,’Lower’)
Select dbo.fn_GenerateText(10,’Upper’)
Select dbo.fn_GenerateText(10,’Mixed’)–default value
Select dbo.fn_GenerateText(10,’Proper’)