Introduction to Memory-Optimized Tables

Memory-optimized tables are fully durable by default, and transactions on memory-optimized tables are fully atomic, consistent, isolated, and durable (ACID).

·      Memory-optimized tables reside in memory.

·      Rows in the table are read from and written to memory. The entire table resides in memory.

·      A second copy of the table data is maintained on disk, but only for durability purposes.

In-Memory OLTP is integrated with SQL Server to provide a seamless experience in all areas such as development, deployment, manageability, and supportability.

Rows in memory-optimized tables are versioned so each row in the table potentially has multiple versions. Row versioning is used to allow concurrent reads and writes on the same row.

The following figure illustrates multi-versioning. The table has three rows: R1, R2, and R3. R1 has three versions, R2 has two versions, and R3 has four versions.

Note: Different versions of the same row do not necessarily occupy consecutive memory locations. Instead, it can be dispersed throughout the table data structure.

Durability

·       Memory-optimized tables are fully durable by default, and they are fully atomic, consistent, isolated, and durable (ACID) (like transactions disk-based tables).

·       In-Memory OLTP supports durable tables with transaction durability delayed. Delayed durable transactions are saved to disk soon after the transaction has committed.

·       Besides the default durable memory-optimized tables, SQL Server also supports non-durable memory-optimized tables, which are not logged, and their data is not persisted on disk. This means that transactions on these tables do not require any disk IO, but the data will not be recovered if there is a server crash or failover.

Durability options:

·       SCHEMA_AND_DATA (default): This option ensures that data is recovered to the Memory-Optimized table when SQL Server is restarted or is recovering from a crash.

·       SCHEMA_ONLY: Like Tempdb data, the SCHEMA_ONLY bound Memory-Optimized table will be truncated if/when SQL Server is restarted or is recovering from a crash, but unlike the tables in Tempdb, the Memory-Optimized table will be re-created as a blank table at the end of the restart/recovery operation

Accessing Data in Memory-Optimized Tables

Data in memory-optimized tables is accessed in two ways:

·       Through interpreted Transact-SQL (outside of a natively-compiled stored procedure). These Transact-SQL statements may be either inside interpreted stored procedures or they may be ad-hoc Transact-SQL statements.

·       Through natively compiled stored procedures.

Performance and Scalability

The following factors will affect the performance gains that can be achieved with In-Memory OLTP:

·       Communication - an application with many calls to short stored procedures may see a smaller performance gain compared to an application with fewer calls and more functionality implemented in each stored procedure.

·       Transact-SQL Execution - In-Memory OLTP achieves the best performance when using natively compiled stored procedures rather than interpreted stored procedures or query execution. Stored procedures that execute other stored procedures cannot be natively compiled, but there can be a benefit to accessing memory-optimized tables from such stored procedures.

·       Range Scan Vs Point Lookup - Memory-optimized nonclustered indexes support range scans and ordered scans. For point lookups, memory-optimized hash indexes have better performance than memory-optimized nonclustered indexes. Memory-optimized nonclustered indexes have better performance than disk-based indexes. Index operations are not logged and they exist only in memory.

·       Concurrency - Applications whose performance is affected by engine-level concurrency, such as latch contention or blocking, improves significantly when the application moves to In-Memory OLTP.

Creating Memory-Optimized Tables

Assumption for below exercise is that you have SQL server 2014 installed. Below examples will be using TestDB database for this exercise.

·       Step 1: Create Memory Optimized File Group

USE [master]

GO

ALTER DATABASE [TestDB] ADD FILEGROUP [fgMOD] CONTAINS MEMORY_OPTIMIZED_DATA

GO

ALTER DATABASE [TestDB] ADD FILE ( NAME = N'SQL2014_MOD', FILENAME = N'D:\MSSQL\SQL2014_MOD' ) TO FILEGROUP [fgMOD]

You can make changes in file name or path as per your requirement.

·       Step 2: Create Memory Optimized Table

Creating an In-Memory (Memory-Optimized) tables in SQL Server is straight forward.

CREATE TABLE dbo.MemoryOptimizedTable

(

tableId INT NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (bucket_count = 2000000)

,naturalId INT NOT NULL INDEX NC_MemoryOptimizedTable_NaturalId NONCLUSTERED HASH WITH (bucket_count = 1000000)

,value NVARCHAR(100) NULL

)

WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA)

GO

Indexing Memory-Optimized Tables

Memory Optimized tables support two types of indexes:

·       Non-Clustered Hash Index: This is a memory optimized index, it does not support inequality operators as well as sort-order matching operations.

·       Non-Clustered Index: This is a disk based index, fully supports all normal index operations.

There are a few stipulations for creating indexes on Memory-Optimized tables:

·       Only 8 indexes are allowed on a Memory Optimized table

·       Indexes cannot be added to a Memory Optimized table, instead the table has to be dropped and re-created with the new index.

·       Primary Key is a requirement for Memory Optimized tables.

·       All indexes are covering, which means they include all columns in a table.

·       Indexes reference the (hashed) row directly, rather than referencing the Primary Key.

Querying Memory-Optimized Tables

Memory-Optimized tables supports access through T-SQL and Natively Compiled Stored Procedures. Generally, you will require a SNAPSHOT isolation level or higher in order to access a Memory-Optimized table. 

References:

• docs.microsoft.com
• thinknook.com

T-SQL code to parse Numbers from a String

Someone asked me recently about T-SQL function to parse numbers from a string. Since there is no built-in function as such, I explained him the logic to calculate same thing using T-SQL code. I will explain the logic here, in case you need to implement same thing (or somewhat similar thing) in your project.

Let’s take example to understand the problem.

Input	Output
Hari12 Sharma34	1234
My 1st code is D$34A and 2nd code is E#078!!!	1342078
Hari# Sharma
$1#0?0 !?@1	1001

Here is the code to get the desired output:

DECLARE @Str VARCHAR(100) = 'My 1st code is D$34A and 2nd code is E#078!!!'

DECLARE @Position SMALLINT = 0

SET @Position = PATINDEX('%[^0-9]%', @Str) --Find first character

 

WHILE (@Position > 0)

BEGIN

    -- Replace alphabet with empty string.

    SET @Str = STUFF(@Str, @Position, 1, '')

   

    -- Find next alphabet

    SET @Position = PATINDEX('%[^0-9]%', @Str)

END

SELECT @Str Result

 

Result

-------------------------------------

1342078

 

We can also write function to encapsulate this code and use wherever required.

This above code can be optimized by using Tally table mentioned below:

DECLARE @Str VARCHAR(100) = 'My 1st code is D$34A and 2nd code is E#078!!!'

 

DECLARE @Result VARCHAR(100)

SELECT  @Result = COALESCE(@Result, '') + Digit

FROM    (

        SELECT  SUBSTRING(@Str, t.ID, 1) AS Digit

        FROM    dbo.Tally t

        WHERE   t.ID <= LEN(@Str)

        AND     PATINDEX('%[0-9]%', SUBString(@Str, t.ID, 1)) = 1

        ) stg

 

SELECT @Result Result

Result

-------------------------------------

1342078

Date-Range multiple tables using T-SQL

Did you ever come across a situation where you need to join multiple tables (consisting FromDate and ToDate) and combine the data into single target table with proper date range?

In this blog post, I will try to explain how to date-range two source tables into single table based on FromDate and ToDate.

Problem: There are two tables Address and Email. Address of any customer can change at any point of time. Likewise, Email ID of customer can be changed at time. We need to track this information using FromDate (start time) and ToDate (end time).

Below are the sample data of these tables:

Address
AddressID	CustomerID	FullAddress	FromDate	ToDate
1	1	148th Ave NE, A10	2005-07-01	2006-08-07
2	1	156th Ave NE, D5	2006-08-08	2009-11-20
3	1	1100 NE 40th ST, A10	2009-11-21	2011-03-31
4	1	148th Ave NE, J101	2011-04-01	9999-12-31
5	2	14528 NE 40th ST	2007-05-01	2008-04-15
6	2	140th Ave NE, K11	2008-04-16	2012-06-25
7	2	145th Ave NE 31st ST	2012-06-26	9999-12-31
Email
EmailID	CustomerID	EmailAddress	FromDate	ToDate
1	1	hari11@gmail.com	2005-07-01	2010-09-30
2	1	hari21@yahoo.com	2010-10-01	2012-12-31
3	1	hari31@hotmail.com	2013-01-01	9999-12-31
4	2	john42@gmail.com	2007-05-01	2013-03-31
5	2	john52@hotmail.com	2013-04-01	9999-12-31

Now consider the above scenario and data, we need following output from these two tables:

Output
CustomerID	FullAddress	EmailAddress	FromDate	ToDate
1	148th Ave NE, A10	hari11@gmail.com	2005-07-01	2006-08-07
1	156th Ave NE, D5	hari11@gmail.com	2006-08-08	2009-11-20
1	1100 NE 40th ST, A10	hari11@gmail.com	2009-11-21	2010-09-30
1	1100 NE 40th ST, A10	hari21@yahoo.com	2010-10-01	2011-03-31
1	148th Ave NE, J101	hari21@yahoo.com	2011-04-01	2012-12-31
1	148th Ave NE, J101	hari31@hotmail.com	2013-01-01	9999-12-31
2	14528 NE 40th ST	john42@gmail.com	2007-05-01	2008-04-15
2	140th Ave NE, K11	john42@gmail.com	2008-04-16	2012-06-25
2	145th Ave NE 31st ST	john42@gmail.com	2012-06-26	2013-03-31
2	145th Ave NE 31st ST	john52@hotmail.com	2013-04-01	9999-12-31

Solution: There could be several ways to achieve the above output. Here is one the T-SQL scripts to get the desired output:
----------------------------------------------------------
--Description: Script to combine different date-ranged
--             columns into single date-ranged table.
----------------------------------------------------------
  
----------------------------------------------------------
--Create test tables
----------------------------------------------------------
IF OBJECT_ID('tempdb..#Address', 'U') IS NOT NULL DROP TABLE #Address
CREATE TABLE #Address
(
     AddressID INT IDENTITY(1, 1)
      ,CustomerID INT NOT NULL
      ,FullAddress VARCHAR(100) NOT NULL
      ,FromDate DATE NOT NULL
      ,ToDate DATE NOT NULL
      ,CONSTRAINT PKC_Address PRIMARY KEY CLUSTERED
   (CustomerID, FromDate)
)
  
IF OBJECT_ID('tempdb..#Email', 'U') IS NOT NULL DROP TABLE #Email
CREATE TABLE #Email
(
     EmailID INT IDENTITY(1, 1)
      ,CustomerID INT NOT NULL
      ,EmailAddress VARCHAR(50) NOT NULL
      ,FromDate DATE NOT NULL
      ,ToDate DATE NOT NULL
      ,CONSTRAINT PKC_Email PRIMARY KEY CLUSTERED
    (CustomerID, FromDate)
)
  
----------------------------------------------------------
--Insert test data
----------------------------------------------------------
INSERT #Address (CustomerID, FullAddress, FromDate, ToDate) VALUES
 (1, '148th Ave NE, A10', '2005-07-01', '2006-08-07')
,(1, '156th Ave NE, D5', '2006-08-08', '2009-11-20')
,(1, '1100 NE 40th ST, A10', '2009-11-21', '2011-03-31')
,(1, '148th Ave NE, J101', '2011-04-01', '9999-12-31')
,(2, '14528 NE 40th ST', '2007-05-01', '2008-04-15')
,(2, '140th Ave NE, K11', '2008-04-16', '2012-06-25')
,(2, '145th Ave NE 31st ST', '2012-06-26', '9999-12-31')

　
INSERT #Email (CustomerID, EmailAddress, FromDate, ToDate) VALUES
 (1, 'hari11@gmail.com', '2005-07-01', '2010-09-30')
,(1, 'hari21@yahoo.com', '2010-10-01', '2012-12-31')
,(1, 'hari31@hotmail.com', '2013-01-01', '9999-12-31')
,(2, 'john42@gmail.com', '2007-05-01', '2013-03-31')
,(2, 'john52@hotmail.com', '2013-04-01', '9999-12-31')

----------------------------------------------------------
--Verify test data
----------------------------------------------------------
SELECT * FROM #Address
SELECT * FROM #Email


----------------------------------------------------------
--T-SQL code to get the required output
----------------------------------------------------------
SELECT    t1.CustomerID
              ,t1.FullAddress
              ,t1.EmailAddress
              ,t1.FromDate
              ,t1.ToDate
FROM
(
SELECT    a.CustomerID
              ,a.FullAddress
              ,e.EmailAddress
              ,(SELECT MAX(U.FromDate)
           FROM (VALUES (a.FromDate), (e.FromDate)) AS U (FromDate)
                ) AS FromDate
               ,(SELECT MIN(U.ToDate)
           FROM (VALUES (a.ToDate), (e.ToDate)) AS U (ToDate)
                ) AS ToDate
FROM      #Address AS a
LEFT JOIN    #Email AS e
ON        a.CustomerID = e.CustomerID
) AS t1
WHERE ISNULL(t1.FromDate, '9999-12-31') <= ISNULL(t1.ToDate, '9999-12-31')

GO