Suppress the Error Number, Severity Level and State Number in the Error output

For any reason, if you don’t want to show the Error Number, Severity Level and the State Number along with the Error Message, use the Severity Level 0 (Zero) or 10. The Severity Number 10 is converted to Zero internally.

Demonstration:

A severity number except 0 or 10 displays the Error Number, Severity and State information.

SuppressErrorNumber1

A severity number 0 or 10 suppresses the Error Number, Severity and State information.

SuppressErrorNumber2

Further Reading:

Database Engine Error Severities
http://msdn.microsoft.com/en-us/library/ms164086(v=sql.105).aspx

How to Use SSIS to Export Clickable URLs to Excel

When SSIS is used to export data to a MS Excel workbook, Excel tries to guess the data type from the first row of data. (Tip: This can be overridden by the IMEX=n parameter in the Excel connection string, where n is the number of rows you want it to scan). As it happens, Excel can not always win the guessing game (or maybe it just plays it safe) and treats data as plain text. Sometimes even numbers and URLs are treated as text and an apostrophe is appended at the beginning of the data value.

If you have a requirement to export hot (i.e. clickable) URLs to Excel, you can use the workaround that I am going to demonstrate below.

We will use the following SQL table. The sample data is intentionally in bad format. We will fix it with a query –

create table URLDemo
(
 id int identity
,url varchar(100)
,friendlyname varchar(100)
);

insert into URLDemo
(url, friendlyname)
values
 ('www.google.com', 'Google')
,('http://www.yahoo.com', 'Yahoo')
,('bing.com', NULL);
-- Data in table ------------------------
select id, url, friendlyname 
from URLDemo

-- Clean data with query ----------------
select
id,
convert(varchar(100),
case
when left(url, 4) = 'www.' then 'http://'+url
when left(url, 11) <> 'http://www.' then 'http://www.'+url
when left(url, 7) <> 'http://' then 'http://'+url
else url
end
) as url,
convert(varchar(100),
case
when isnull(friendlyname, '') = '' then url
else friendlyname
end
) as friendlyname
from URLDemo

ExcelURLDemo1

The Clean Data query is an attempt to add any missing http://www. to the URL and also show a friendly name if missing. These will be useful later for the Excel HYPERLINK formula.

The problem –

We can export this data to Excel, even try to encode the Excel HYPERLINK formula as a derived column, the url and the formula will show up just as plain text, that is they will not be clickable.

The workaround –

Create an Excel Workbook that will be the template for data export. Type in the column names [id], [url] and [friendlyname] for column A, B and C respectively. Put the following Excel formula in the column D :-

=IF( AND(B1<>"", C1<>""), HYPERLINK(B1,C1), "")

Drag this formula cell to the number of rows as appropriate for your expected data volume. I did it for the first 20 rows. The idea here is to populate the [url] and [friendlyname] columns via SSIS and let the formula create a clickable link for us.

ExcelURLDemo2

Then select cells A1 to C1 and give this range a name, e.g. MyDataRange.

ExcelURLDemo3

Now create an SSIS package. Add a Data Flow Task and double-click to open its designer. Add a OLEDB Source in the Data Flow designer. The query in this editor is the Clean Data query above.

ExcelURLDemo4

Add a Data Conversion Task. It will be used to convert non-unicode (VARCHAR) columns namely [url] and [friendlyname] to unicode. Excel likes to have character data in unicode.

ExcelURLDemo5

Add an Excel Destination. It will use the data range name. Verify the column mappings.

ExcelURLDemo6

Gyaan: Excel tries to identify the last “dirty” row in the worksheet and puts new data from SSIS in the next row. In my case it will be row 21 because I dragged the formula till row 20. Even if the cells once used are emptied by deleting the values/formulas, they are still considered “dirty”. The only way to remove the “dirty” flag from a cell is to delete the whole row! If the package will put data in row 21 and beyond, we miss out on the formulas that we put in the first 20 rows. Using the range tells Excel to look for dirty cells only within the range, thus allowing us to use the formulas in the adjacent cells.

Now verify the column mappings. Note that Excel columns [url] and [friendlyname] are mapped to the [urlWSTR] and [friendlynameWSTR] of the Data Conversion.

ExcelURLDemo7

That is it, execute the package and click away at the URLs!

ExcelURLDemo8

Example of SSIS Foreach Loop Item Enumerator and Dynamic OLEDB Connection

Foreach Loop Container is one of the looping constructs available in SSIS packages. One of its enumerator options is the Foreach Item Enumerator. It enumerates through a list of items populated at design time. The list is static at execution time, i.e. you cannot load the list items dynamically from a configuration file or another source like an SQL table. It is probably due to the static nature of the item list, that this enumerator type is not used too often in packages.

On the other hand, there are some scenarios where a static list can prove to be useful. For example –

  • no need to create a database table just to pass on values for the Foreach Loop
  • enforce a specific order to the parameters passed to the loop
  • initialize variables values with the Item Enumerator instead of doing that in a Script task

Another frequent requirement in SSIS packages is to execute a set of operations against different SQL Servers or databases. The package should change the OLE DB connection string dynamically during execution to point towards the appropriate target. A looping construct iterates through the list of servers and databases, and the package Expressions change the OLE DB connection properties.

I am creating this step-by-step scenario to demonstrate the use of a Foreach Item Enumerator and use it to make a dynamic OLEDB connection. I will execute a single query in a loop against three different SQL databases and export the results to a single text file.

The Steps

Add a Foreach Loop Container to your SSIS package. Double-click on it to open its editor. Click on the Collections tab on the left of the editor. Select the enumerator as Foreach Item Enumerator. Click on the Columns… button. Add two columns of String type. As you add columns, the editor gives them names like Column0, Column1 and so on, and there is no way to customize these names.

ItemEnumeratorDemo1

Populate the column values by simply typing in the rows. Use Column0 for server names and Column1 for corresponding database names on that server. I have used my local server for all three rows but you can use your own server names here. This is the list of items that are iterated by the Foreach Loop.

ItemEnumeratorDemo2

Now you need two variables that will hold the values shredded from the item list as the Foreach Loop iterates through the rows. These variables are used later in the Property Expressions of the OLEDB connection to make it dynamic.

Click on Variable Mappings tab on the left and then on the <New Variable…>

ItemEnumeratorDemo3

Add two variables of String type and name them ServerName and DatabaseName.

ItemEnumeratorDemo4

The Index 0 and 1 of the variables correspond to the Column0 and Column1 of the Foreach Item list respectively.

ItemEnumeratorDemo5

That is all there is to it. Now the loop is ready to iterate through the item rows.

Let us move on to the dynamic OLEDB connection. Add a Data Flow Task to the Foreach Loop Container.

ItemEnumeratorDemo6

Double-click the Data Flow Task to view its design surface. Now add one OLE DB Source.

ItemEnumeratorDemo7

Double-click OLE DB Source to set its properties. The first window is of its connection manager.

ItemEnumeratorDemo8

Once the connection information is done, view its properties editor. Set the Data Access Mode to SQL Command and put the following query in the SQL Command Text –

select
@@SERVERNAME as ServerName
,DB_NAME() as DatabaseName
,GETDATE() as LogDateTime
,COUNT(*) as NumberOfIndexes
from sys.indexes;

ItemEnumeratorDemo9

Clicking the Columns tab is important so that the SSIS can find the metadata like column names, data types etc. from the underlying tables/views. If you don’t do this then you’ll find later that SSIS does not know of this metadata.

ItemEnumeratorDemo10

Again, it is a good idea to drag the connector from OLEDB source to the Flat File destination before modifying the Flat File destination properties. In this way, the SSIS Designer helps you by matching the destination with the source metadata automatically. Otherwise, you have to perform some extra steps to match columns. I hope you like to save time as much as I do!

ItemEnumeratorDemo11

Give a file path and check the box for Column Names in the First Row.

ItemEnumeratorDemo12

Verify the column mappings in the Mappings tab.

ItemEnumeratorDemo13

The trick to make the OLEDB connection string dynamic lies in the next couple of steps where you edit the properties of the OLEDB connection and assign expressions to its ServerName and InitialCatalog properties.

ItemEnumeratorDemo14

The property ServerName gets the variable @[User::ServerName] and the property InitialCatalog gets the variable @[User::DatabaseName]. Remember that these variables will get their values modified during each iteration of the Foreach Loop and as a result, the OLEDB connection string will get modified too.

ItemEnumeratorDemo15

As the last step, make sure that data will append to the contents of the destination text file. Otherwise, each iteration will overwrite the contents with the new source connection.

ItemEnumeratorDemo16

Find Top 25 Inefficient Query Plans by CPU, IO, Recompiles, Execution Count

Microsoft TechNet Gallery is a treasure trove of scripts that can save you a lot of coding time or sometimes introduce creative ways of solving a challenge.

I came across this collection of scripts that will show you the top 25 inefficient query plans (in XML format) sorted by CPU, IO, recompiles, execution counts etc.

Download: http://gallery.technet.microsoft.com/Find-inefficient-query-88f4611f

SSIS Components Naming Conventions

Read a New Version of This Blog Post – SSIS Naming Convention – Updated for SSIS 2016

I wrote earlier about my naming conventions for SSIS variables that let me quickly figure out the source of the variable value very easily. This post continues on to the prefix-based naming conventions I use (or plan to use) for the 120+ components in SSIS packages.

While looking around online for SSIS component naming conventions, most of the suggestions are based on Jamie Thompson’s (Blog | @JamieT) post SSIS: Suggested Best Practices and naming conventions. He has tried to limit the prefixes to 3 or 4 characters and included most of the frequently used components. Using the prefixes as described in his post has helped me a lot in the maintenance phase of a package when my memory has faded about the functionality and the workflow. Some more motivation to adopt a naming convention can be found in 31-days of SSIS series by Jason Strate (Blog | @StrateSQL).

My goal was to make an exhaustive list of components with their prefixes so that –

  • I do not need to put on my thinking hat every time I start using a component for the first time.
  • The naming convention can be future-proof and avoid confusion with new components being introduced in SSIS platform.
  • Remove the guesswork (especially for new developers and maintenance teams who may not be SSIS developers primarily) by avoiding prefixes that sound too similar to another.

I created an Excel sheet of all the components (more than 120) that I found listed in MSDN, i.e. containers, connection managers, sources, destinations, tasks, and transformations. My prefixes are more elaborate than Jamie’s for the reasons mentioned above, but that is just my preference. You can use the Excel as a template for you organization and modify the prefixes to you liking.

The following is an example of one section from the Excel.

Download: Get the full Excel workbook at the TechNet Galleries

Example –

SSIS Component Naming Conventions

Spring Time!

Keeping up with the spirit of spring season, I’ve refreshed the look of the blog by using the “Flounder” theme. I was using the “Enterprise” theme earlier.

What I like about Flounder is :

  • the colour scheme
  • the overall font choice
  • a well-defined sidebar
  • and most of all, its mobile device compatibility. It readjusts from 3-column layout to a 2-column or a 1-column layout as the screen size becomes smaller. That really allows very easy browsing on a tablet or a cell phone, for folks who don’t use the WordPress app, by completely avoiding horizontal scrolling.

I have also changed the blog’s tag line from “Learnings as a SQL Server DBA and Developer” to “Erudition as a SQL DBA and Developer”.

er·u·di·tion [er-yoo-dish-uhn, er-oo-]

noun
knowledge acquired by study, research, etc.; learning; scholarship.

And finally, I’ve added my Twitter handle @TsqlREX to the publicize section of the blog so that a new post is announced on Twitter too. I’m not a big Twitter user right now but I plan to learn more about it in this year.

[jetpack_subscription_form subscribe_button=”Sign Me Up”]

TSQL Gotcha – Putting a Condition in WHERE or ON clause: Does it Matter in TSQL?

A filter condition can be syntactically put in the WHERE clause of the query or the ON clause too. But does it matter?

No, it does not matter for an INNER join. I found this one example of an exception in Martin Smith’s comment on Stack Overflow where using the GROUP BY ALL makes a difference in results. GROUP BY ALL is now a deprecated syntax (Feature ID 169) so probably we can ignore this exception. But it is always good to know.

On the other hand, Yes, it could affect the output in any of the OUTER joins.

This happens because when a filter criteria is made a part of the JOIN condition, it is applied at the time of reading the tables to identify the rows for the JOIN. The rows that do not satisfy the filter criteria are not included for the purpose of the JOIN and that causes NULLs in the OUTER JOINs.

However, with the filter criteria a part of the WHERE clause, it is applied after the query has been processed and the adhoc dataset has been created internally. The final dataset is reduced to just those rows which meet the filter criteria.

I will use a criteria of c.isactiveYN = ‘Y’ for demonstration below. Rows with a value of ‘Y’ are used for the join. The rows with ‘N’ or NULLs don’t match the criteria so they are left out.

As the examples show below, this seemingly minor difference can cause substantial difference in the query output of OUTER JOINs. This effect could cascade on to other subsequent tables in bigger queries that are joined to the original tables.

-- Create demo tables and insert some data
declare @customer table 
(id int, 
cname varchar(20), 
isactiveYN char(1))

declare @order table 
(id int, 
customerid int, 
orderdate date)

insert into @customer 
values
(1, 'Bob',   'Y'),
(2, 'Harry', 'N'),
(3, 'Max',   'Y')

insert into @order
values
(1, 1, '2014-1-1'),
(2, 1, '2014-2-2'),
(3, 2, '2014-3-3'),
(4, 2, '2014-4-4'),
(5, 3, '2014-5-5'),
(6, 3, '2014-6-6')

select * from @customer
select * from @order

Results :

FilterPositionDifference-1-DemoTables

 

Demonstration: The location of criteria has no effect on the results in INNER JOIN.

-- A simple INNER JOIN without any 
-- conditions to display all data:

-- INNER JOIN with out any conditions
select * 
from 
  @customer as c
inner join 
  @order as o
  on c.id = o.customerid

-- The following two queries differ
-- in the location of the criteria
-- but still show similar results

-- INNER JOIN with filter in WHERE clause
select * 
from 
  @customer as c
inner join 
  @order as o
  on c.id = o.customerid
where
  c.isactiveYN = 'Y'

-- INNER JOIN with filter in ON clause
select * 
from
  @customer as c
inner join 
  @order as o
  on c.id = o.customerid
  and c.isactiveYN = 'Y'

Results : The second and third results are similar.

FilterPositionDifference-2-InnerJoinExamples

 

Demonstration: LEFT OUTER joins with the condition in the WHERE clause and the ON clause

-- LEFT JOIN: Filter in WHERE clause
select * 
from 
  @customer as c
left outer join 
  @order as o
  on c.id = o.customerid
where 
  c.isactiveYN = 'Y'

-- LEFT JOIN: Filter in ON clause
select * 
from 
  @customer as c
left outer join
  @order as o
  on c.id = o.customerid
  and c.isactiveYN = 'Y'

Results : “Harry” is not in the first result-set because the row was filtered out by the WHERE clause. “Harry” is in the second result-set because it comes from the LEFT OUTER table but there is no matching row because it was not used for the JOIN.

FilterPositionDifference-3-LeftJoin

 

Demonstration: RIGHT OUTER joins with the condition in the WHERE clause and the ON clause

-- RIGHT JOIN: Filter in WHERE clause
select * 
from 
  @customer as c
right outer join 
  @order as o
  on c.id = o.customerid
where 
  c.isactiveYN = 'Y'

-- RIGHT JOIN: Filter in ON clause
select * 
from 
  @customer as c
right outer join 
  @order as o
  on c.id = o.customerid
  and c.isactiveYN = 'Y'

Results : “Harry” is not in the first result-set because the row was filtered out by the WHERE clause. “Harry” is not in the second result-set because it has been filtered out and it is also not in the OUTER table.

FilterPositionDifference-4-RightJoin

 

Summary: When the criteria is in the WHERE clause, The results of the INNER, LEFT OUTER and the RIGHT OUTER queries are the same. Differences show up in the results of LEFT and RIGHT OUTER queries when the criteria is in the ON clause.

Folks, Give Your Eyes a Break

I work long hours looking at computer monitor(s). On certain days, the eye-strain used to get too heavy on my eyes and I couldn’t even look at people straight in the eye!

I needed some kind of reminder to pause, look away from the monitors, and maybe take a water break.

There are many programs that show cute pop-up reminders. But they are on the same screen that I’m trying to look away from in the first place! Also most of them are easy to ignore. I didn’t want a system lock screen because logging in frequently becomes an irritant after some time.

I’ve started using the good old Windows Task Scheduler along with the NirCmd utility by NirSoft. It is a simple EXE file and does not need any installation. While it can do a lot of useful stuff but I was interested in just turning the monitor on/off. Especially good if you work on laptops that don’t have dedicated on/off switch for monitors.

  • NirCmd.exe concise documentation and download page (scroll to the bottom to see both 32-bit and 64-bit versions).
  • NirCmd.exe extensive documentation page.

I extracted the 64-bit zip in my Downloads folder and created a simple batch file with the following commands –

CD "C:UsersAalam.RangiDownloadsnircmd-x64"
nircmd.exe speak text "Turning monitor Off in 5 seconds"
timeout 5
nircmd.exe monitor off
timeout 5
nircmd.exe speak text "Turning monitor on in 30 seconds"
timeout 30
nircmd.exe monitor on

Yes, it speaks too! Then I opened the Windows Task Scheduler and scheduled the batch file as a 30 minute job.

There you go! It turns off my monitor every 30 minutes for 30 seconds. I get up from my chair for a mini stretch, take a gulp of water, look at the farthest end of the room or maybe out of the window, while the monitor turns back on.

Now there could be some folks whose workplace enterprise policy won’t let them download an EXE file from Internet. You could use a screensaver instead. E.g. the following line in the batch file will trigger the Mystify screensaver. A reminder enough for the break. Also, if it is just going to be a single command then maybe you don’t even need a batch file. You can put this command in the scheduled task by itself.

@START /WAIT /MIN %windir%system32mystify.scr -s

The downside is that this job keeps running even after I’ve left my office. Might startle the janitor 🙂

TIMESTAMP and ROWVERSION datatypes and the Behavior of the SQL Server @@DBTS Counter

The TIMESTAMP datatype in SQL Server is in no way related to any Date/Time datatype or any operation related to dates or time. It is a simple monotonously incrementing counter that increases by one every time there is an INSERT or UPDATE operation in the database. It is stored as BINARY but can be converted to BIGINT or INT type to make the values seem more user-friendly. The IDENTITY property of a table is quite similar when we think about a monotonously incrementing value. While the IDENTITY counter is tracked at a table level, the TIMESTAMP counter is tracked at a database level. The current value of the TIMESTAMP counter can be returned by using the Configration Function @@DBTS. We can assume a TIMESTAMP to be a GUID in the smaller context of a database. So theoretically no two tables in a database can have the same TIMESTAMP value in any of their rows. But practically it is possible to go around this via SELECT..INTO syntax to create a new table, as I’ll demostrate later in this post.

As calling it a TIMESTAMP has caused confusion among SQL developers, Microsoft introduced ROWVERSION as a new name for it. Calling it ROWVERSION also emphasizes its utility for implementing optimistic locking in the code. TIMESTAMP has been deprecated and will be discontinued in a future SQL Server release so use ROWVERSION in all new development work.

It is optional to give the column a name in the DDL with the TIMESTAMP datatype. If a column name is not specified then SQL Server uses the default name of “TIMESTAMP” for the column. Helpful, eh?

On the other hand, if using the ROWVERSION as datatype, a column name must be given just like creating a column any other datatype.

A table can have only one column of TIMESTAMP (or ROWVERSION) type.

You cannot enter/update a value explicitly into a column of TIMESTAMP or ROWVERSION datatype. The database manages it internally. If you do try then an error is returned –

Msg 273, Level 16, State 1, Line 1

Cannot insert an explicit value into a timestamp column. Use INSERT with a column list to exclude the timestamp column, or insert a DEFAULT into the timestamp column.

The DBTS value starts from 2000 in a brand new database without any table containing a TIMESTAMP/ROWVERSION column.

CREATE DATABASE [TestDB] 
GO

-- Find the default value of 
-- the TIMESTAMP/ROWVERSION counter 
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS AS BIGINT) AsBigint 

/* Result
AsBinary                AsBigint 
0x00000000000007D0      2000 
*/

Does the value of the counter change after new table is created with a TIMESTAMP/ROWVERSION?

No. Just creating a table does not change the counter.

CREATE TABLE a (
       id INT IDENTITY, 
       TIMESTAMP)
GO
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS AS BIGINT) AsBigint 
/* Result
AsBinary               AsBigint 
0x00000000000007D0     2000 
*/

Insert a row in a table and check counter value. It increases by one.

INSERT INTO a DEFAULT VALUES
GO
SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM a 

SELECT @@DBTS AsBinary, 
       CAST(@@DBTS  AS BIGINT) AsBigint 
/* Result
AsBinary               AsBigint 
0x00000000000007D1     2001 
*/

Create a new table, this time with ROWVERSION and insert a row. The counter still increases by one.

CREATE TABLE b (
       id INT IDENTITY, 
       col2  ROWVERSION) 
GO
INSERT INTO b DEFAULT VALUES
GO
SELECT *, 
       CAST(col2  AS BIGINT) 
FROM b 

-- Find the value of the counter after 
-- a row is inserted 
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS  AS BIGINT) AsBigint 
/* Result
AsBinary               AsBigint 
0x00000000000007D2     2002 
*/

Do the INSERTS in the same transaction have the same counter value? Let us try.

BEGIN TRAN 
INSERT INTO a DEFAULT VALUES
INSERT INTO b DEFAULT VALUES
COMMIT

SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM a 

SELECT *, 
       CAST(col2 AS BIGINT) 
FROM b 

-- Find the value of the counter after rows 
-- are inserted in multiple tables in a transaction. 
-- Each row still gets a unique counter value.
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS  AS BIGINT) AsBigint 
/* Result
AsBinary               AsBigint 
0x00000000000007D4     2004 
*/

Does the counter increase in case of a failed transaction?

BEGIN TRAN 
INSERT INTO a DEFAULT VALUES
INSERT INTO b DEFAULT VALUES
ROLLBACK

SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM a 

SELECT *, 
       CAST(col2 AS BIGINT) 
FROM b 

-- Find the counter value after a failed transaction
-- Yes. The counter value still increases.
SELECT @@DBTS AsBinary, 
CAST(@@DBTS AS BIGINT) AsBigint 
/* Result
AsBinary               AsBigint 
0x00000000000007D6     2006 
*/

Is is possible to have duplicate TIMESTAMP values across tables of a database?
Yes. Although values in TIMESTAMP/ROWVERSION columns cannot be explicitly inserted or modified they can be copied from one table to another using the SELECT..INTO syntax, which in turn could result in duplicate counter values across tables

SELECT *
INTO c -- New table
FROM a

SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM a -- Source table 

SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM c -- New table 

-- The counter value does not change.
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS AS BIGINT) AsBigint 
/*
AsBinary               AsBigint 
0x00000000000007D6     2006 
*/

Does the counter continue with insertions into new table that was created with SELECT…INTO? Yes, the counter changes because the new table gets the column definition of the source table including the ROWVERSION column.

INSERT INTO c DEFAULT VALUES
SELECT *, 
       CAST(timestamp AS BIGINT) 
FROM c -- New table 

-- Find the value of the counter after a row 
-- is inserted into the new table.
-- The counter value does change.
SELECT @@DBTS AsBinary, 
       CAST(@@DBTS AS BIGINT) AsBigint 
/*
AsBinary               AsBigint 
0x00000000000007D7     2007 
*/

Final review of the counter values in tables

SELECT *, CAST(timestamp AS BIGINT) 
FROM a 

SELECT *, CAST(col2 AS BIGINT) 
FROM b 

SELECT *, CAST(timestamp AS BIGINT) 
FROM c