<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>SQL Fool &#187; large</title>
	<atom:link href="http://sqlfool.com/tag/large/feed/" rel="self" type="application/rss+xml" />
	<link>http://sqlfool.com</link>
	<description>Self-Professed SQL Scripting Junkie!</description>
	<lastBuildDate>Wed, 02 May 2012 21:25:28 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	
		<item>
		<title>BCP Basics</title>
		<link>http://sqlfool.com/2008/12/bcp-basics/</link>
		<comments>http://sqlfool.com/2008/12/bcp-basics/#comments</comments>
		<pubDate>Mon, 01 Dec 2008 17:20:59 +0000</pubDate>
		<dc:creator>Michelle Ufford</dc:creator>
				<category><![CDATA[Performance & Tuning]]></category>
		<category><![CDATA[SQL Tips]]></category>
		<category><![CDATA[bcp]]></category>
		<category><![CDATA[large]]></category>

		<guid isPermaLink="false">http://sqlfool.com/?p=222</guid>
		<description><![CDATA[In this blog post, I&#8217;m going to walk through the basics of BCP (bulk copy program). BCP is a utility that installs with SQL Server and can assist with large data transfers. Let&#8217;s see what parameter options are available to use. From the command line on a machine with SQL Server installed, type &#8220;bcp&#8221; and [...]]]></description>
			<content:encoded><![CDATA[<p>In this blog post, I&#8217;m going to walk through the basics of BCP (bulk copy program).  BCP is a utility that installs with SQL Server and can assist with large data transfers.</p>
<p>Let&#8217;s see what parameter options are available to use.  From the command line on a machine with SQL Server installed, type &#8220;bcp&#8221; and press Enter.</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_1.jpg"><img title="BCP Parameters" src="http://sqlfool.com/blogImages/20081130/BCP_1_small.jpg" alt="BCP Parameters" /></a></p>
<p>You can find out more information on BCP parameters on Books Online: <a href="http://msdn.microsoft.com/en-us/library/ms162802.aspx" target="_blank">http://msdn.microsoft.com/en-us/library/ms162802.aspx</a></p>
<p>For now, we&#8217;re going to examine just the basics.  The simplest syntax of a BCP command is:</p>
<p>bcp<br />
databaseName.Schema.TableName *or* &#8220;Query&#8221;<br />
in, out, *or* queryout<br />
-S ServerName\instanceName<br />
-U userName -P password *or* -T<br />
-c *or* -n *or* specify storage information for each column</p>
<p>Let&#8217;s look at these options in a little more detail:</p>
<p><em>databaseName.Schema.TableName *or* Query</em><br />
You can specify either an entire table to copy or a query.  The query should be surrounded in quotations and must also include the fully qualified table name.</p>
<p><em>in, out, *or* queryout</em><br />
in = import, out = full table export, queryout = query to select data for export</p>
<p><em>-U userName -P password *or* -T</em><br />
You can either specify a specific account to access SQL Server, or use -T to indicate Trusted Connection (i.e. Windows Authentication)</p>
<p><em>-c *or* -n *or* specify storage information for each column</em><br />
-c indicates character data type, -n indicates native data type; if neither one is specified, by default you will be prompted for the data type for each column.</p>
<p>Now let&#8217;s put this together and run some BCP commands.  All of these examples will use the AdventureWorks 2008 sample database.</p>
<p>First, let&#8217;s export an entire table.  To do this, we&#8217;ll use the &#8220;out&#8221; parameter.</p>

<div class="wp_syntax"><div class="code"><pre class="text" style="font-family:monospace;">bcp AdventureWorks.Sales.SalesOrderDetail out
C:\bcp_outputTable.txt -SYourServerName -T -c</pre></div></div>

<p>&nbsp;</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_2.jpg"><img title="Export Table with BCP" src="http://sqlfool.com/blogImages/20081130/BCP_2_small.jpg" alt="Export Table with BCP" /></a></p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_3.jpg"><img title="Export Table with BCP - Results" src="http://sqlfool.com/blogImages/20081130/BCP_3_small.jpg" alt="Export Table with BCP - Results" /></a></p>
<p>I don&#8217;t normally export an entire table&#8230; or at least, not in one process.  So let&#8217;s walk through what it would look like to export the same table using a query.  This will use the &#8220;queryout&#8221; parameter.</p>
<p>&nbsp;</p>

<div class="wp_syntax"><div class="code"><pre class="text" style="font-family:monospace;">bcp &quot;Select SalesOrderID, SalesOrderDetailID, OrderQty, ProductID
From AdventureWorks.Sales.SalesOrderDetail&quot; queryout
C:\bcp_outputQuery.txt -SYourServerName -T -c</pre></div></div>

<p>&nbsp;</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_4.jpg"><img title="Export Query with BCP" src="http://sqlfool.com/blogImages/20081130/BCP_4_small.jpg" alt="Export Query with BCP" /></a></p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_5.jpg"><img title="Export Query with BCP - Results" src="http://sqlfool.com/blogImages/20081130/BCP_5_small.jpg" alt="Export Query with BCP - Results" /></a></p>
<p>You&#8217;ll notice that the total duration for the query was shorter than for the full-table export.  This is because we&#8217;re only exporting a few of the columns.  This is important to keep in mind when bcp&#8217;ing data: you&#8217;ll get better performance if you only export the data elements that you actually need.</p>
<p>Now that we&#8217;ve exported some data, let&#8217;s walk through the process of importing this data.  First, let&#8217;s create a table with a constraint that will result in some errors.</p>

<div class="wp_syntax"><div class="code"><pre class="tsql" style="font-family:monospace;"><span style="color: #0000FF;">Create</span> <span style="color: #0000FF;">Table</span> dbo.<span style="color: #202020;">testBCPLoad</span>
<span style="color: #808080;">&#40;</span>
      SalesOrderID          <span style="color: #0000FF;">int</span>      Not Null
    , SalesOrderDetailID    <span style="color: #0000FF;">int</span>      Not Null
    , OrderQty              <span style="color: #0000FF;">smallint</span> Null
    , ProductID             <span style="color: #0000FF;">int</span>      Null
&nbsp;
    <span style="color: #0000FF;">Constraint</span> PK_testBCPLoad
        <span style="color: #0000FF;">Primary</span> <span style="color: #0000FF;">Key</span> <span style="color: #0000FF;">Clustered</span>
        <span style="color: #808080;">&#40;</span>SalesOrderID<span style="color: #808080;">&#41;</span>
<span style="color: #808080;">&#41;</span>;</pre></div></div>

<p>Now execute the BCP import command:</p>

<div class="wp_syntax"><div class="code"><pre class="text" style="font-family:monospace;">bcp sandbox.dbo.testBCPLoad in
C:\bcp_outputQuery.txt -SYourServername -T -c</pre></div></div>

<p>&nbsp;</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_6.jpg"><img title="Load Data with BCP" src="http://sqlfool.com/blogImages/20081130/BCP_6_small.jpg" alt="Load Data with BCP" /></a></p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_7.jpg"><img title="Load Data with BCP - Error" src="http://sqlfool.com/blogImages/20081130/BCP_7_small.jpg" alt="Load Data with BCP - Error" /></a></p>
<p>You should receive a Primary Key error.  When you check your results in SQL Server, you should find no results loaded into the table.  This is BCP&#8217;s default behavior.</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_8.jpg"><img title="Check Destination Table" src="http://sqlfool.com/blogImages/20081130/BCP_8_small.jpg" alt="Check Destination Table" /></a></p>
<p>Let&#8217;s change our constraint and try the same BCP command again:</p>

<div class="wp_syntax"><div class="code"><pre class="tsql" style="font-family:monospace;"><span style="color: #0000FF;">Alter</span> <span style="color: #0000FF;">Table</span> dbo.<span style="color: #202020;">testBCPLoad</span>
    <span style="color: #0000FF;">Drop</span> <span style="color: #0000FF;">Constraint</span> PK_testBCPLoad;
&nbsp;
<span style="color: #0000FF;">Alter</span> <span style="color: #0000FF;">Table</span> dbo.<span style="color: #202020;">testBCPLoad</span>
    <span style="color: #0000FF;">Add</span> <span style="color: #0000FF;">Constraint</span> PK_testBCPLoad
    <span style="color: #0000FF;">Primary</span> <span style="color: #0000FF;">Key</span> <span style="color: #0000FF;">Clustered</span>
        <span style="color: #808080;">&#40;</span>SalesOrderID, SalesOrderDetailID<span style="color: #808080;">&#41;</span>;</pre></div></div>

<p>&nbsp;</p>

<div class="wp_syntax"><div class="code"><pre class="text" style="font-family:monospace;">bcp sandbox.dbo.testBCPLoad in
C:\bcp_outputQuery.txt -SYourServername -T -c</pre></div></div>

<p>You should now have the data loaded into your SQL Server destination table:</p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_9.jpg"><img title="Import Data with BCP - Results" src="http://sqlfool.com/blogImages/20081130/BCP_9_small.jpg" alt="Import Data with BCP - Results" /></a></p>
<p><a href="http://sqlfool.com/blogImages/20081130/BCP_10.jpg"><img title="Destination Table" src="http://sqlfool.com/blogImages/20081130/BCP_10_small.jpg" alt="Destination Table" /></a></p>
<p>So there you have it, the basics of BCP!  <img src='http://sqlfool.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>A few BCP tips:</p>
<ul>
<li>BCP commands are case-sensitive!</li>
<li>If you&#8217;re accessing the data across a WAN, perhaps via a VPN connection, try to remote desktop (mstsc) to the actual SQL Server to perform the BCP.  If possible, keep the operation on the same local drive or even local network as the server; the less distance data needs to travel across a network, the faster BCP will perform.</li>
<li>If you need to copy large amounts of data (i.e. &gt;100mm rows), try breaking the data into smaller chunks.  This will help if you have an error during BCP (i.e. a PK error can rollback the entire import operation by default, although there are options that can change this behavior).  When working with partitioned tables, I find it very efficient to segregate the data imported/exported by partition.</li>
<li>If you&#8217;re BCP&#8217;ing data into a new table, you can minimize impact on the server by waiting to create your indexes after all the data is loaded.</li>
<li>I like to construct my queries in SSMS, then copy them to BCP.  Since the command-line utility does not support copy and pasting, I create a text file with my BCP command in NotePad, then save the command as a .cmd.  To execute, just call the .cmd file.</li>
</ul>
]]></content:encoded>
			<wfw:commentRss>http://sqlfool.com/2008/12/bcp-basics/feed/</wfw:commentRss>
		<slash:comments>25</slash:comments>
		</item>
		<item>
		<title>Bulk Inserts with XML</title>
		<link>http://sqlfool.com/2008/11/bulk-inserts-with-xml/</link>
		<comments>http://sqlfool.com/2008/11/bulk-inserts-with-xml/#comments</comments>
		<pubDate>Wed, 19 Nov 2008 19:24:38 +0000</pubDate>
		<dc:creator>Michelle Ufford</dc:creator>
				<category><![CDATA[Performance & Tuning]]></category>
		<category><![CDATA[SQL Tips]]></category>
		<category><![CDATA[T-SQL Scripts]]></category>
		<category><![CDATA[large]]></category>
		<category><![CDATA[performance]]></category>
		<category><![CDATA[script]]></category>
		<category><![CDATA[XML]]></category>

		<guid isPermaLink="false">http://sqlfool.com/?p=194</guid>
		<description><![CDATA[Last week, I blogged about how to perform one-to-many inserts with table-valued parameters, a feature new in 2008. For those who do not yet have 2008 or will not have it in the near future, it may still be beneficial to use XML for bulk inserts. Here&#8217;s a pretty simple example of how to accomplish [...]]]></description>
			<content:encoded><![CDATA[<p>Last week, I blogged about <a href="http://sqlfool.com/2008/11/one-to-many-inserts-with-table-valued-parameters/" target="_blank">how to perform one-to-many inserts with table-valued parameters</a>, a feature new in 2008.  For those who do not yet have 2008 or will not have it in the near future, it may still be beneficial to use XML for bulk inserts.</p>
<p>Here&#8217;s a pretty simple example of how to accomplish this:</p>
<p>
<div class="wp_syntax"><div class="code"><pre class="tsql" style="font-family:monospace;"><span style="color: #008080;">/* Create some tables to work with */</span>
<span style="color: #0000FF;">CREATE</span> <span style="color: #0000FF;">TABLE</span> dbo.<span style="color: #202020;">orders</span>
<span style="color: #808080;">&#40;</span>
      order_id      <span style="color: #0000FF;">INT</span> <span style="color: #0000FF;">IDENTITY</span><span style="color: #808080;">&#40;</span><span style="color: #000;">1</span>,<span style="color: #000;">1</span><span style="color: #808080;">&#41;</span>   Not Null
    , orderDate     <span style="color: #0000FF;">DATETIME</span>            Not Null
    , customer_id   <span style="color: #0000FF;">INT</span>                 Not Null
&nbsp;
    <span style="color: #0000FF;">CONSTRAINT</span> PK_orders
        <span style="color: #0000FF;">PRIMARY</span> <span style="color: #0000FF;">KEY</span> <span style="color: #0000FF;">CLUSTERED</span><span style="color: #808080;">&#40;</span>order_id<span style="color: #808080;">&#41;</span>
<span style="color: #808080;">&#41;</span>;
&nbsp;
<span style="color: #0000FF;">CREATE</span> <span style="color: #0000FF;">TABLE</span> dbo.<span style="color: #202020;">orderDetails</span>
<span style="color: #808080;">&#40;</span>
      orderDetail_id    <span style="color: #0000FF;">INT</span> <span style="color: #0000FF;">IDENTITY</span><span style="color: #808080;">&#40;</span><span style="color: #000;">1</span>,<span style="color: #000;">1</span><span style="color: #808080;">&#41;</span>   Not Null
    , order_id          <span style="color: #0000FF;">INT</span>                 Not Null
    , lineItem          <span style="color: #0000FF;">INT</span>                 Not Null
    , product_id        <span style="color: #0000FF;">INT</span>                 Not Null
&nbsp;
    <span style="color: #0000FF;">CONSTRAINT</span> PK_orderDetails
        <span style="color: #0000FF;">PRIMARY</span> <span style="color: #0000FF;">KEY</span> <span style="color: #0000FF;">CLUSTERED</span><span style="color: #808080;">&#40;</span>orderDetail_id<span style="color: #808080;">&#41;</span>
&nbsp;
    <span style="color: #0000FF;">CONSTRAINT</span> FK_orderDetails_orderID
        <span style="color: #0000FF;">FOREIGN</span> <span style="color: #0000FF;">KEY</span><span style="color: #808080;">&#40;</span>order_id<span style="color: #808080;">&#41;</span>
        <span style="color: #0000FF;">REFERENCES</span> dbo.<span style="color: #202020;">orders</span><span style="color: #808080;">&#40;</span>order_id<span style="color: #808080;">&#41;</span>
<span style="color: #808080;">&#41;</span>;
Go
&nbsp;
<span style="color: #008080;">/* Create a new procedure using an XML parameter */</span>
<span style="color: #0000FF;">CREATE</span> <span style="color: #0000FF;">PROCEDURE</span> dbo.<span style="color: #202020;">insert_orderXML_sp</span>
      @orderDate        <span style="color: #0000FF;">DATETIME</span>
    , @customer_id      <span style="color: #0000FF;">INT</span>
    , @orderDetailsXML  XML
<span style="color: #0000FF;">AS</span>
<span style="color: #0000FF;">BEGIN</span>
&nbsp;
    <span style="color: #0000FF;">SET</span> <span style="color: #0000FF;">NOCOUNT</span> <span style="color: #0000FF;">ON</span>;
&nbsp;
    <span style="color: #0000FF;">DECLARE</span> @myOrderID <span style="color: #0000FF;">INT</span>;
&nbsp;
    <span style="color: #0000FF;">INSERT</span> <span style="color: #0000FF;">INTO</span> dbo.<span style="color: #202020;">orders</span>
    <span style="color: #808080;">&#40;</span>
          orderDate
        , customer_id    
    <span style="color: #808080;">&#41;</span>
    <span style="color: #0000FF;">VALUES</span>
    <span style="color: #808080;">&#40;</span>
          @orderDate
        , @customer_id
    <span style="color: #808080;">&#41;</span>;
&nbsp;
    <span style="color: #0000FF;">SET</span> @myOrderID <span style="color: #808080;">=</span> <span style="color: #FF00FF;">SCOPE_IDENTITY</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>;
&nbsp;
    <span style="color: #0000FF;">INSERT</span> <span style="color: #0000FF;">INTO</span> dbo.<span style="color: #202020;">orderDetails</span>
    <span style="color: #808080;">&#40;</span>
          order_id
        , lineItem
        , product_id
    <span style="color: #808080;">&#41;</span>
    <span style="color: #0000FF;">SELECT</span> @myOrderID
         , myXML.<span style="color: #0000FF;">value</span><span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'./@lineItem'</span>, <span style="color: #FF0000;">'int'</span><span style="color: #808080;">&#41;</span>
         , myXML.<span style="color: #0000FF;">value</span><span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'./@product_id'</span>, <span style="color: #FF0000;">'int'</span><span style="color: #808080;">&#41;</span>
    <span style="color: #0000FF;">FROM</span> @orderDetailsXML.<span style="color: #202020;">nodes</span><span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'/orderDetail'</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">As</span> nodes<span style="color: #808080;">&#40;</span>myXML<span style="color: #808080;">&#41;</span>;
&nbsp;
    <span style="color: #0000FF;">SET</span> <span style="color: #0000FF;">NOCOUNT</span> <span style="color: #0000FF;">OFF</span>;
&nbsp;
<span style="color: #0000FF;">END</span>
GO
&nbsp;
&nbsp;
<span style="color: #008080;">/* Call our stored procedure */</span>
<span style="color: #0000FF;">EXECUTE</span> dbo.<span style="color: #202020;">insert_orderXML_sp</span>
      @orderDate <span style="color: #808080;">=</span> <span style="color: #FF0000;">'2008-01-01'</span>
    , @customer_id <span style="color: #808080;">=</span> <span style="color: #000;">101</span>
    , @orderDetailsXML <span style="color: #808080;">=</span> 
        <span style="color: #FF0000;">'&lt;orderDetail lineItem=&quot;1&quot; product_id=&quot;123&quot; /&gt;
         &lt;orderDetail lineItem=&quot;2&quot; product_id=&quot;456&quot; /&gt;
         &lt;orderDetail lineItem=&quot;3&quot; product_id=&quot;789&quot; /&gt;
         &lt;orderDetail lineItem=&quot;4&quot; product_id=&quot;246&quot; /&gt;
         &lt;orderDetail lineItem=&quot;5&quot; product_id=&quot;135&quot; /&gt;'</span>;
&nbsp;
&nbsp;
<span style="color: #008080;">/* Check our data */</span>
<span style="color: #0000FF;">SELECT</span> <span style="color: #808080;">*</span> <span style="color: #0000FF;">FROM</span> dbo.<span style="color: #202020;">orders</span>;
<span style="color: #0000FF;">SELECT</span> <span style="color: #808080;">*</span> <span style="color: #0000FF;">FROM</span> dbo.<span style="color: #202020;">orderDetails</span>;
&nbsp;
&nbsp;
<span style="color: #008080;">/* Clean up our mess */</span>
<span style="color: #0000FF;">DROP</span> <span style="color: #0000FF;">PROCEDURE</span> insert_orderXML_sp;
<span style="color: #0000FF;">DROP</span> <span style="color: #0000FF;">TABLE</span> dbo.<span style="color: #202020;">orderDetails</span>;
<span style="color: #0000FF;">DROP</span> <span style="color: #0000FF;">TABLE</span> dbo.<span style="color: #202020;">orders</span>;</pre></div></div>

</p>
<p>
I&#8217;ve found that this is more efficient when performing large parent/child inserts, i.e. 1 parent record to 100 child records.  Keep in mind that there&#8217;s a point where doing an insert with XML is more expensive than using a traditional INSERT stored procedure.  I haven&#8217;t run any tests yet to help define what that threshold is&#8230; more to come on this in the near future.</p>
<p>More on the Nodes() Method can be found here in Books Online: <a href="http://msdn.microsoft.com/en-us/library/ms188282(SQL.90).aspx" target="_blank">http://msdn.microsoft.com/en-us/library/ms188282(SQL.90).aspx</a></p>
<p><strong>Update:</strong>  I&#8217;ve just learned that the &#8220;value&#8221; keyword is case-sensitive.  Apparently my code box plug-in was defaulting &#8220;value&#8221; to &#8220;VALUE.&#8221;  <img src='http://sqlfool.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>Here&#8217;s the error message you&#8217;ll get if you don&#8217;t have &#8220;value&#8221; in lower-case:<br />
<span style="color: #ff0000;">Cannot find either column &#8220;myXML&#8221; or the user-defined function or aggregate &#8220;myXML.VALUE&#8221;, or the name is ambiguous.</span></p>
]]></content:encoded>
			<wfw:commentRss>http://sqlfool.com/2008/11/bulk-inserts-with-xml/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Performance Comparison of Singleton, XML, and TVP Inserts</title>
		<link>http://sqlfool.com/2008/11/performance-comparison-of-singleton-xml-and-tvp-inserts/</link>
		<comments>http://sqlfool.com/2008/11/performance-comparison-of-singleton-xml-and-tvp-inserts/#comments</comments>
		<pubDate>Fri, 14 Nov 2008 01:26:33 +0000</pubDate>
		<dc:creator>Michelle Ufford</dc:creator>
				<category><![CDATA[Performance & Tuning]]></category>
		<category><![CDATA[SQL 2008]]></category>
		<category><![CDATA[SQL Tips]]></category>
		<category><![CDATA[2008]]></category>
		<category><![CDATA[large]]></category>
		<category><![CDATA[performance]]></category>

		<guid isPermaLink="false">http://sqlfool.com/?p=156</guid>
		<description><![CDATA[As promised, today I took a look at the performance of bulk inserts using XML and Table-Valued Parameters. I also compared it against singleton inserts to show the value in the bulk-insert approach. My tests were pretty simple: insert 100 records using each method. Each test was executed 10 times to ensure consistency. The duration [...]]]></description>
			<content:encoded><![CDATA[<p>As promised, today I took a look at the performance of bulk inserts using XML and Table-Valued Parameters.  I also compared it against singleton inserts to show the value in the bulk-insert approach.</p>
<p>My tests were pretty simple: insert 100 records using each method.  Each test was executed 10 times to ensure consistency.  The duration was recorded in microseconds.  </p>
<p>The goal was to compare the performance of the inserts.  Because I was executing this entire test within SQL Server, I had to isolate only the actual insert transactions and ignore everything else, such as the loading of the data; that work would normally be performed by the calling application.</p>
<p>So without further ado&#8230; screenshots of the Profiler traces: (click to enlarge)</p>
<div class="wp-caption alignnone" style="width: 753px"><a href="http://www.sqlfool.com/blogImages/20081113/SingletonInsert.gif"><img alt="TVP" src="http://www.sqlfool.com/blogImages/20081113/SingletonInsert.gif" title="Single Insert Method" width="400" height="200" /></a><p class="wp-caption-text">Single Insert Method</p></div>
<div class="wp-caption alignnone" style="width: 753px"><a href="http://www.sqlfool.com/blogImages/20081113/XML.gif"><img alt="TVP" src="http://www.sqlfool.com/blogImages/20081113/XML.gif" title="XML Method" width="400" height="200" /></a><p class="wp-caption-text">XML Method</p></div>
<div class="wp-caption alignnone" style="width: 753px"><a href="http://www.sqlfool.com/blogImages/20081113/TVP.gif"><img alt="TVP" src="http://www.sqlfool.com/blogImages/20081113/TVP.gif" title="Table-Valued Parameter Method" width="400" height="200" /></a><p class="wp-caption-text">Table-Valued Parameter Method</p></div>
<p><strong>Summary</strong></p>
<table border="1" cellspacing="0" cellpadding="5">
<tr>
<td>Method</td>
<td>Avg CPU</td>
<td>Avg Reads</td>
<td>Avg Writes</td>
<td>Avg Duration (micro)</td>
</tr>
<tr>
<td>Singleton Method</td>
<td>3</td>
<td>202</td>
<td>0</td>
<td>13378</td>
</tr>
<tr>
<td>XML Method</td>
<td>0</td>
<td>222</td>
<td>0</td>
<td>3124</td>
</tr>
<tr>
<td>TVP Method</td>
<td>1</td>
<td>207</td>
<td>0</td>
<td>780</td>
</tr>
</table>
<p>&nbsp;</p>
<p>As expected, both the XML and the TVP method performed significantly better than the single-insert method.  As hoped, the table-valued parameter arguably performed the best of all 3. </p>
]]></content:encoded>
			<wfw:commentRss>http://sqlfool.com/2008/11/performance-comparison-of-singleton-xml-and-tvp-inserts/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Large Updates on Replicated Tables</title>
		<link>http://sqlfool.com/2008/11/large-updates-on-replicated-tables/</link>
		<comments>http://sqlfool.com/2008/11/large-updates-on-replicated-tables/#comments</comments>
		<pubDate>Sat, 08 Nov 2008 17:42:35 +0000</pubDate>
		<dc:creator>Michelle Ufford</dc:creator>
				<category><![CDATA[SQL Tips]]></category>
		<category><![CDATA[T-SQL Scripts]]></category>
		<category><![CDATA[large]]></category>
		<category><![CDATA[replication]]></category>
		<category><![CDATA[script]]></category>
		<category><![CDATA[TSQL]]></category>
		<category><![CDATA[update]]></category>

		<guid isPermaLink="false">http://sqlfool.com/?p=145</guid>
		<description><![CDATA[Late last night, I executed a 70mm update on a replicated table. This was a somewhat low priority update, so the primary goal (aside from the data change) was to avoid any issues on the publishing and subscribing servers, including replication latency errors. I have performed many large updates in the past, but this was [...]]]></description>
			<content:encoded><![CDATA[<p>Late last night, I executed a 70mm update on a replicated table.  This was a somewhat low priority update, so the primary goal (aside from the data change) was to avoid any issues on the publishing and subscribing servers, including replication latency errors.  I have performed many large updates in the past, but this was the first on a replicated table.  </p>
<p>To minimize impact, one of our system DBA&#8217;s suggested the use of a replicated stored procedure.  Basically, instead of replicating each of the 70mm updates as a singleton transaction, the replicated stored procedure is called on the subscribing server, which then performs the bulk update locally.  This was my first time using it and it worked beautifully.  </p>
<p>Another of my colleagues, Jeff M., suggested the use of a control table.  Normally, I would just output the last affected ID and update the script manually.  However, this also worked so well that it will be adopted in all future update scripts.</p>
<p>Using the following pseudo-script and a replicated stored procedure, I was able to execute the update on 70mm records in 11.5 hours with *zero* impact on the servers and replication.  I could&#8217;ve probably increased the batch size and reduced the execution time even further, but as I mentioned, this was a low priority update, so there was no need to push it.</p>
<p>This process should work equally well on non-replicated updates; merely replace the Execute statement with the actual update.</p>

<div class="wp_syntax"><div class="code"><pre class="tsql" style="font-family:monospace;"><span style="color: #008080;">/************ Chunked Update Script with Control Table ************/</span>
&nbsp;
<span style="color: #008080;">/* --------------------- Preparation Script --------------------- */</span>
&nbsp;
<span style="color: #008080;">/* Note: Execute this section in a separate query window */</span>
&nbsp;
<span style="color: #0000FF;">Use</span> yourDatabase;
Go
&nbsp;
<span style="color: #0000FF;">Set</span> <span style="color: #0000FF;">NoCount</span> <span style="color: #0000FF;">On</span>;
&nbsp;
<span style="color: #008080;">/* Create a control table to facilitate tweaking of parameters */</span>
<span style="color: #0000FF;">Create</span> <span style="color: #0000FF;">Table</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span>
<span style="color: #808080;">&#40;</span>
      sizeOfBatch       <span style="color: #0000FF;">int</span>
    , waitForDelay      <span style="color: #0000FF;">char</span><span style="color: #808080;">&#40;</span><span style="color: #000;">8</span><span style="color: #808080;">&#41;</span>
    , minRecordToUpdate <span style="color: #0000FF;">int</span> 
    , maxRecordToUpdate <span style="color: #0000FF;">int</span>
<span style="color: #808080;">&#41;</span>;
&nbsp;
<span style="color: #008080;">/* Create your control data; you only want 1 row in this table */</span>
<span style="color: #0000FF;">Insert</span> <span style="color: #0000FF;">Into</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span> 
<span style="color: #808080;">&#40;</span>sizeOfBatch, waitForDelay, minRecordToUpdate, maxRecordToUpdate<span style="color: #808080;">&#41;</span>
<span style="color: #0000FF;">Select</span> <span style="color: #000;">10000</span>, <span style="color: #FF0000;">'00:00:05'</span>, <span style="color: #000;">40297132</span>, <span style="color: #000;">107459380</span>;
&nbsp;
<span style="color: #008080;">/* Update Script */</span>
<span style="color: #0000FF;">Update</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span>
<span style="color: #0000FF;">Set</span> sizeOfBatch  <span style="color: #808080;">=</span> <span style="color: #000;">100000</span>
  , waitForDelay <span style="color: #808080;">=</span> <span style="color: #FF0000;">'00:00:30'</span>;
&nbsp;
&nbsp;
<span style="color: #008080;">/* ------------------------ Update Script ------------------------ */</span>
<span style="color: #0000FF;">Use</span> yourDatabase;
Go
&nbsp;
<span style="color: #0000FF;">Set</span> <span style="color: #0000FF;">NoCount</span> <span style="color: #0000FF;">On</span>;
&nbsp;
<span style="color: #0000FF;">Declare</span> 
      @batchSize        <span style="color: #0000FF;">int</span>
    , @minID            <span style="color: #0000FF;">int</span>
    , @maxID            <span style="color: #0000FF;">int</span>
    , @procMinID        <span style="color: #0000FF;">int</span>
    , @procMaxID        <span style="color: #0000FF;">int</span>
    , @delay            <span style="color: #0000FF;">char</span><span style="color: #808080;">&#40;</span><span style="color: #000;">8</span><span style="color: #808080;">&#41;</span>
    , @statusMsg        <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">1000</span><span style="color: #808080;">&#41;</span>;
&nbsp;
<span style="color: #0000FF;">Begin</span> Try
&nbsp;
    <span style="color: #0000FF;">IF</span> <span style="color: #FF00FF;">@@SERVERNAME</span> Not In <span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'PRODSERVER'</span><span style="color: #808080;">&#41;</span>
        <span style="color: #0000FF;">RAISERROR</span><span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'Sorry, this cannot be executed here!'</span>, <span style="color: #000;">16</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span>;
&nbsp;
    <span style="color: #0000FF;">If</span> Not Exists<span style="color: #808080;">&#40;</span>
            <span style="color: #0000FF;">Select</span> <span style="color: #FF00FF;">object_id</span> 
            <span style="color: #0000FF;">From</span> sys.<span style="color: #202020;">objects</span> 
            <span style="color: #0000FF;">Where</span> <span style="color: #808080;">&#91;</span>name<span style="color: #808080;">&#93;</span> <span style="color: #808080;">=</span> <span style="color: #FF0000;">'scratch_largeUpdate_control'</span> 
            And type <span style="color: #808080;">=</span> <span style="color: #FF0000;">'U'</span> <span style="color: #808080;">&#41;</span>
        <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'ERROR: Control table does not exist!'</span>, <span style="color: #000;">16</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span>
            <span style="color: #0000FF;">With</span> NoWait;
    <span style="color: #0000FF;">Else</span>
        <span style="color: #0000FF;">Select</span> 
              @minID        <span style="color: #808080;">=</span> minRecordToUpdate 
            , @maxID        <span style="color: #808080;">=</span> maxRecordToUpdate 
            , @batchSize    <span style="color: #808080;">=</span> sizeOfBatch
            , @delay        <span style="color: #808080;">=</span> waitForDelay
        <span style="color: #0000FF;">From</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span> <span style="color: #0000FF;">With</span> <span style="color: #808080;">&#40;</span>NoLock<span style="color: #808080;">&#41;</span>;
&nbsp;
    <span style="color: #0000FF;">Set</span> @statusMsg <span style="color: #808080;">=</span> <span style="color: #FF0000;">'Beginning update; batch size of '</span> 
        <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span>@batchSize <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> <span style="color: #808080;">+</span> <span style="color: #FF0000;">', delay of '</span> 
        <span style="color: #808080;">+</span> @delay <span style="color: #808080;">+</span> <span style="color: #FF0000;">' defined.  Estimate '</span> 
        <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#40;</span>@maxID <span style="color: #808080;">-</span> @minID<span style="color: #808080;">&#41;</span> <span style="color: #808080;">/</span> @batchSize<span style="color: #808080;">&#41;</span> <span style="color: #808080;">+</span> <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> 
        <span style="color: #808080;">+</span> <span style="color: #FF0000;">' iterations to be performed.'</span>
&nbsp;
    <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span>@statusMsg, <span style="color: #000;">10</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">With</span> NoWait;
&nbsp;
    <span style="color: #0000FF;">While</span> @minID <span style="color: #808080;">&lt;</span> @maxID
    <span style="color: #0000FF;">Begin</span>
&nbsp;
        <span style="color: #0000FF;">Select</span> @procMinID <span style="color: #808080;">=</span> @minID
            , @procMaxID <span style="color: #808080;">=</span> <span style="color: #808080;">&#40;</span>@minID <span style="color: #808080;">+</span> <span style="color: #808080;">&#40;</span>@batchSize <span style="color: #808080;">-</span> <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span>;
&nbsp;
        <span style="color: #008080;">/* Execute actual update code here 
           OR 
           Call a replicated stored procedure, i.e. */</span>
        <span style="color: #0000FF;">Execute</span> dbo.<span style="color: #202020;">myReplicatedUpdateProc</span> 
              @minRecordID <span style="color: #808080;">=</span> @procMinID
            , @maxRecordID <span style="color: #808080;">=</span> @procMaxID;
&nbsp;
        <span style="color: #0000FF;">Set</span> @statusMsg <span style="color: #808080;">=</span> 
            <span style="color: #FF0000;">'Updating records '</span> <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span>@minID <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> 
            <span style="color: #808080;">+</span> <span style="color: #FF0000;">' through '</span> <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#40;</span>@minID <span style="color: #808080;">+</span> <span style="color: #808080;">&#40;</span>@batchSize <span style="color: #808080;">-</span> <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> 
            <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> <span style="color: #808080;">+</span> <span style="color: #FF0000;">'...'</span>;
&nbsp;
        <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span>@statusMsg, <span style="color: #000;">10</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">With</span> NoWait;
&nbsp;
        <span style="color: #008080;">/* Update our control table with the last successfully
           updated record ID.  In the event of an error,
           we can start from here. */</span>
        <span style="color: #0000FF;">Update</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span> 
        <span style="color: #0000FF;">Set</span> minRecordToUpdate <span style="color: #808080;">=</span> @minID <span style="color: #808080;">+</span> @batchSize;
&nbsp;
        <span style="color: #0000FF;">Select</span> @minID <span style="color: #808080;">=</span> @minID <span style="color: #808080;">+</span> @batchSize; 
        <span style="color: #0000FF;">WaitFor</span> Delay @delay; <span style="color: #008080;">-- breather for the server</span>
&nbsp;
        <span style="color: #008080;">/* Check to see if our control values have changed */</span>
        <span style="color: #0000FF;">If</span> Not Exists<span style="color: #808080;">&#40;</span>
            <span style="color: #0000FF;">Select</span> <span style="color: #808080;">*</span> 
            <span style="color: #0000FF;">From</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span> <span style="color: #0000FF;">With</span> <span style="color: #808080;">&#40;</span>NoLock<span style="color: #808080;">&#41;</span> 
            <span style="color: #0000FF;">Where</span> @batchSize <span style="color: #808080;">=</span> sizeOfBatch And @delay <span style="color: #808080;">=</span> waitForDelay<span style="color: #808080;">&#41;</span>
        <span style="color: #0000FF;">Begin</span>
&nbsp;
            <span style="color: #008080;">/* There was a change, so grab our new values */</span>
            <span style="color: #0000FF;">Select</span> @batchSize <span style="color: #808080;">=</span> sizeOfBatch
                 , @delay <span style="color: #808080;">=</span> waitForDelay
            <span style="color: #0000FF;">From</span> dbo.<span style="color: #202020;">scratch_largeUpdate_control</span> <span style="color: #0000FF;">With</span> <span style="color: #808080;">&#40;</span>NoLock<span style="color: #808080;">&#41;</span>
&nbsp;
            <span style="color: #008080;">/* Print a status message with the new values */</span>
            <span style="color: #0000FF;">Set</span> @statusMsg <span style="color: #808080;">=</span> <span style="color: #FF0000;">'Parameters changed:  batch size = '</span> 
                <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span>@batchSize <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> 
                <span style="color: #808080;">+</span> <span style="color: #FF0000;">', delay = '</span> <span style="color: #808080;">+</span> @delay;
&nbsp;
            <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span>@statusMsg, <span style="color: #000;">10</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">With</span> NoWait;
&nbsp;
        <span style="color: #0000FF;">End</span>
    <span style="color: #0000FF;">End</span>
&nbsp;
    <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span><span style="color: #FF0000;">'Success!'</span>, <span style="color: #000;">10</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">With</span> NoWait;
&nbsp;
<span style="color: #0000FF;">End</span> Try
<span style="color: #008080;">/* Handle your errors */</span>
<span style="color: #0000FF;">Begin</span> Catch
&nbsp;
        <span style="color: #0000FF;">Set</span> @statusMsg <span style="color: #808080;">=</span> <span style="color: #FF0000;">'An error has occurred and the last '</span>
                         <span style="color: #808080;">+</span> <span style="color: #FF0000;">'transaction has been rolled back. '</span>
                         <span style="color: #808080;">+</span> <span style="color: #FF0000;">'Last record successfully updated was '</span>
                         <span style="color: #808080;">+</span> <span style="color: #FF0000;">'record_id = '</span> 
                         <span style="color: #808080;">+</span> <span style="color: #0000FF;">Cast</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#40;</span>@minID <span style="color: #808080;">+</span> <span style="color: #808080;">&#40;</span>@batchSize <span style="color: #808080;">-</span> <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span> 
                            <span style="color: #0000FF;">As</span> <span style="color: #0000FF;">varchar</span><span style="color: #808080;">&#40;</span><span style="color: #000;">10</span><span style="color: #808080;">&#41;</span><span style="color: #808080;">&#41;</span>;
&nbsp;
        <span style="color: #0000FF;">RaisError</span> <span style="color: #808080;">&#40;</span>@statusMsg, <span style="color: #000;">16</span>, <span style="color: #000;">1</span><span style="color: #808080;">&#41;</span> <span style="color: #0000FF;">With</span> NoWait;
&nbsp;
        <span style="color: #008080;">/* Return the error message */</span>
        <span style="color: #0000FF;">Select</span> Error_Number<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
            , Error_Procedure<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
		    , <span style="color: #FF00FF;">DB_Name</span><span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
		    , Error_Line<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
		    , Error_Message<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
		    , Error_Severity<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>
		    , Error_State<span style="color: #808080;">&#40;</span><span style="color: #808080;">&#41;</span>;
&nbsp;
<span style="color: #0000FF;">End</span> Catch;
&nbsp;
<span style="color: #008080;">/* -------------------------- Clean-Up ----------------------------
Drop Table dbo.scratch_largeUpdate_control;
----------------------------------------------------------------- */</span></pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://sqlfool.com/2008/11/large-updates-on-replicated-tables/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

