/usr/share/doc/php-doctrine-orm/html/reference/batch-processing.html is in doctrine-orm-doc 2.4.6-1+deb8u1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>14. Batch Processing — Doctrine 2 ORM 2 documentation</title>
<link rel="stylesheet" href="../_static/bootstrap/css/bootstrap.min.css" type="text/css" />
<link rel="stylesheet" href="../_static/default.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/layout.css" type="text/css" />
<link rel="stylesheet" href="../_static/configurationblock.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '2',
COLLAPSE_MODINDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/configurationblock.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/configurationblock.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/configurationblock.js"></script>
<script src="../_static/bootstrap/js/bootstrap.min.js"></script>
<script type="text/javascript">
<!--
$(document).ready(function() {
$("#versions").change(function() {
var docsUrl = $(this).val();
window.location.href = docsUrl;
});
});
-->
</script>
<link rel="shortcut icon" href="../_static/doctrine.ico"/>
<link rel="top" title="Doctrine 2 ORM 2 documentation" href="../index.html" />
<link rel="next" title="15. Doctrine Query Language" href="dql-doctrine-query-language.html" />
<link rel="prev" title="13. Transactions and Concurrency" href="transactions-and-concurrency.html" />
</head>
<body>
<div id="wrapper">
<div id="header">
<h1 id="h1title"></h1>
<div id="logo">
<a href="http://www.doctrine-project.org/">Doctrine - PHP Database Libraries</a>
</div>
</div>
<div id="nav" class="cls">
<div class="tl cls">
<ul>
<li><a target="_top" href="/">home</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/about">about</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/projects.html">projects</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/projects/orm">orm</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/projects/dbal">dbal</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/blog">blog</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/jira">development</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/contribute">contribute</a></li>
<li><a class="" target="_top" href="http://www.doctrine-project.org/community">community</a></li>
</ul>
</div>
</div>
<div id="content" class="cls">
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="dql-doctrine-query-language.html" title="15. Doctrine Query Language"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="transactions-and-concurrency.html" title="13. Transactions and Concurrency"
accesskey="P">previous</a> |</li>
<li><a href="/">Doctrine Homepage</a> »</li>
<li><a href="../index.html">Doctrine 2 ORM 2 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" >
<div class="section" id="batch-processing">
<h1>14. Batch Processing<a class="headerlink" href="#batch-processing" title="Permalink to this headline">¶</a></h1>
<p>This chapter shows you how to accomplish bulk inserts, updates and
deletes with Doctrine in an efficient way. The main problem with
bulk operations is usually not to run out of memory and this is
especially what the strategies presented here provide help with.</p>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">An ORM tool is not primarily well-suited for mass
inserts, updates or deletions. Every RDBMS has its own, most
effective way of dealing with such operations and if the options
outlined below are not sufficient for your purposes we recommend
you use the tools for your particular RDBMS for these bulk
operations.</p>
</div>
<div class="section" id="bulk-inserts">
<h2>14.1. Bulk Inserts<a class="headerlink" href="#bulk-inserts" title="Permalink to this headline">¶</a></h2>
<p>Bulk inserts in Doctrine are best performed in batches, taking
advantage of the transactional write-behind behavior of an
<tt class="docutils literal"><span class="pre">EntityManager</span></tt>. The following code shows an example for
inserting 10000 objects with a batch size of 20. You may need to
experiment with the batch size to find the size that works best for
you. Larger batch sizes mean more prepared statement reuse
internally but also mean more work during <tt class="docutils literal"><span class="pre">flush</span></tt>.</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$batchSize</span> <span class="o">=</span> <span class="mi">20</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nv">$i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="nv">$i</span> <span class="o"><=</span> <span class="mi">10000</span><span class="p">;</span> <span class="o">++</span><span class="nv">$i</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">CmsUser</span><span class="p">;</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">setStatus</span><span class="p">(</span><span class="s1">'user'</span><span class="p">);</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">setUsername</span><span class="p">(</span><span class="s1">'user'</span> <span class="o">.</span> <span class="nv">$i</span><span class="p">);</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">setName</span><span class="p">(</span><span class="s1">'Mr.Smith-'</span> <span class="o">.</span> <span class="nv">$i</span><span class="p">);</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">persist</span><span class="p">(</span><span class="nv">$user</span><span class="p">);</span>
<span class="k">if</span> <span class="p">((</span><span class="nv">$i</span> <span class="o">%</span> <span class="nv">$batchSize</span><span class="p">)</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">flush</span><span class="p">();</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">clear</span><span class="p">();</span> <span class="c1">// Detaches all objects from Doctrine!</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="section" id="bulk-updates">
<h2>14.2. Bulk Updates<a class="headerlink" href="#bulk-updates" title="Permalink to this headline">¶</a></h2>
<p>There are 2 possibilities for bulk updates with Doctrine.</p>
<div class="section" id="dql-update">
<h3>14.2.1. DQL UPDATE<a class="headerlink" href="#dql-update" title="Permalink to this headline">¶</a></h3>
<p>The by far most efficient way for bulk updates is to use a DQL
UPDATE query. Example:</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$q</span> <span class="o">=</span> <span class="nv">$em</span><span class="o">-></span><span class="na">createQuery</span><span class="p">(</span><span class="s1">'update MyProject\Model\Manager m set m.salary = m.salary * 0.9'</span><span class="p">);</span>
<span class="nv">$numUpdated</span> <span class="o">=</span> <span class="nv">$q</span><span class="o">-></span><span class="na">execute</span><span class="p">();</span>
</pre></div>
</div>
</div>
<div class="section" id="iterating-results">
<h3>14.2.2. Iterating results<a class="headerlink" href="#iterating-results" title="Permalink to this headline">¶</a></h3>
<p>An alternative solution for bulk updates is to use the
<tt class="docutils literal"><span class="pre">Query#iterate()</span></tt> facility to iterate over the query results step
by step instead of loading the whole result into memory at once.
The following example shows how to do this, combining the iteration
with the batching strategy that was already used for bulk inserts:</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$batchSize</span> <span class="o">=</span> <span class="mi">20</span><span class="p">;</span>
<span class="nv">$i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">$q</span> <span class="o">=</span> <span class="nv">$em</span><span class="o">-></span><span class="na">createQuery</span><span class="p">(</span><span class="s1">'select u from MyProject\Model\User u'</span><span class="p">);</span>
<span class="nv">$iterableResult</span> <span class="o">=</span> <span class="nv">$q</span><span class="o">-></span><span class="na">iterate</span><span class="p">();</span>
<span class="k">foreach</span><span class="p">(</span><span class="nv">$iterableResult</span> <span class="k">AS</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$row</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">increaseCredit</span><span class="p">();</span>
<span class="nv">$user</span><span class="o">-></span><span class="na">calculateNewBonuses</span><span class="p">();</span>
<span class="k">if</span> <span class="p">((</span><span class="nv">$i</span> <span class="o">%</span> <span class="nv">$batchSize</span><span class="p">)</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">flush</span><span class="p">();</span> <span class="c1">// Executes all updates.</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">clear</span><span class="p">();</span> <span class="c1">// Detaches all objects from Doctrine!</span>
<span class="p">}</span>
<span class="o">++</span><span class="nv">$i</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">flush</span><span class="p">();</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Iterating results is not possible with queries that
fetch-join a collection-valued association. The nature of such SQL
result sets is not suitable for incremental hydration.</p>
</div>
</div>
</div>
<div class="section" id="bulk-deletes">
<h2>14.3. Bulk Deletes<a class="headerlink" href="#bulk-deletes" title="Permalink to this headline">¶</a></h2>
<p>There are two possibilities for bulk deletes with Doctrine. You can
either issue a single DQL DELETE query or you can iterate over
results removing them one at a time.</p>
<div class="section" id="dql-delete">
<h3>14.3.1. DQL DELETE<a class="headerlink" href="#dql-delete" title="Permalink to this headline">¶</a></h3>
<p>The by far most efficient way for bulk deletes is to use a DQL
DELETE query.</p>
<p>Example:</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$q</span> <span class="o">=</span> <span class="nv">$em</span><span class="o">-></span><span class="na">createQuery</span><span class="p">(</span><span class="s1">'delete from MyProject\Model\Manager m where m.salary > 100000'</span><span class="p">);</span>
<span class="nv">$numDeleted</span> <span class="o">=</span> <span class="nv">$q</span><span class="o">-></span><span class="na">execute</span><span class="p">();</span>
</pre></div>
</div>
</div>
<div class="section" id="id1">
<h3>14.3.2. Iterating results<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
<p>An alternative solution for bulk deletes is to use the
<tt class="docutils literal"><span class="pre">Query#iterate()</span></tt> facility to iterate over the query results step
by step instead of loading the whole result into memory at once.
The following example shows how to do this:</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$batchSize</span> <span class="o">=</span> <span class="mi">20</span><span class="p">;</span>
<span class="nv">$i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">$q</span> <span class="o">=</span> <span class="nv">$em</span><span class="o">-></span><span class="na">createQuery</span><span class="p">(</span><span class="s1">'select u from MyProject\Model\User u'</span><span class="p">);</span>
<span class="nv">$iterableResult</span> <span class="o">=</span> <span class="nv">$q</span><span class="o">-></span><span class="na">iterate</span><span class="p">();</span>
<span class="k">while</span> <span class="p">((</span><span class="nv">$row</span> <span class="o">=</span> <span class="nv">$iterableResult</span><span class="o">-></span><span class="na">next</span><span class="p">())</span> <span class="o">!==</span> <span class="k">false</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">remove</span><span class="p">(</span><span class="nv">$row</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="k">if</span> <span class="p">((</span><span class="nv">$i</span> <span class="o">%</span> <span class="nv">$batchSize</span><span class="p">)</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">flush</span><span class="p">();</span> <span class="c1">// Executes all deletions.</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">clear</span><span class="p">();</span> <span class="c1">// Detaches all objects from Doctrine!</span>
<span class="p">}</span>
<span class="o">++</span><span class="nv">$i</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$em</span><span class="o">-></span><span class="na">flush</span><span class="p">();</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Iterating results is not possible with queries that
fetch-join a collection-valued association. The nature of such SQL
result sets is not suitable for incremental hydration.</p>
</div>
</div>
</div>
<div class="section" id="iterating-large-results-for-data-processing">
<h2>14.4. Iterating Large Results for Data-Processing<a class="headerlink" href="#iterating-large-results-for-data-processing" title="Permalink to this headline">¶</a></h2>
<p>You can use the <tt class="docutils literal"><span class="pre">iterate()</span></tt> method just to iterate over a large
result and no UPDATE or DELETE intention. The <tt class="docutils literal"><span class="pre">IterableResult</span></tt>
instance returned from <tt class="docutils literal"><span class="pre">$query->iterate()</span></tt> implements the
Iterator interface so you can process a large result without memory
problems using the following approach:</p>
<div class="highlight-php"><div class="highlight"><pre><span class="cp"><?php</span>
<span class="nv">$q</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="na">_em</span><span class="o">-></span><span class="na">createQuery</span><span class="p">(</span><span class="s1">'select u from MyProject\Model\User u'</span><span class="p">);</span>
<span class="nv">$iterableResult</span> <span class="o">=</span> <span class="nv">$q</span><span class="o">-></span><span class="na">iterate</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$iterableResult</span> <span class="k">AS</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// do stuff with the data in the row, $row[0] is always the object</span>
<span class="c1">// detach from Doctrine, so that it can be Garbage-Collected immediately</span>
<span class="nv">$this</span><span class="o">-></span><span class="na">_em</span><span class="o">-></span><span class="na">detach</span><span class="p">(</span><span class="nv">$row</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Iterating results is not possible with queries that
fetch-join a collection-valued association. The nature of such SQL
result sets is not suitable for incremental hydration.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<div id="searchbox" style="">
<h3>Search</h3>
<form class="search" action="http://readthedocs.org/search/project/" method="get">
<input type="text" name="q" size="18">
<input type="submit" value="Go">
<input type="hidden" name="selected_facets" value="project:">
</form>
</div>
<h3><a href="../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">14. Batch Processing</a><ul>
<li><a class="reference internal" href="#bulk-inserts">14.1. Bulk Inserts</a></li>
<li><a class="reference internal" href="#bulk-updates">14.2. Bulk Updates</a><ul>
<li><a class="reference internal" href="#dql-update">14.2.1. DQL UPDATE</a></li>
<li><a class="reference internal" href="#iterating-results">14.2.2. Iterating results</a></li>
</ul>
</li>
<li><a class="reference internal" href="#bulk-deletes">14.3. Bulk Deletes</a><ul>
<li><a class="reference internal" href="#dql-delete">14.3.1. DQL DELETE</a></li>
<li><a class="reference internal" href="#id1">14.3.2. Iterating results</a></li>
</ul>
</li>
<li><a class="reference internal" href="#iterating-large-results-for-data-processing">14.4. Iterating Large Results for Data-Processing</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="transactions-and-concurrency.html"
title="previous chapter">13. Transactions and Concurrency</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="dql-doctrine-query-language.html"
title="next chapter">15. Doctrine Query Language</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/reference/batch-processing.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
© Copyright 2010-12, Doctrine Project Team.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.2.3.
<br/>
<a target="_BLANK" href="http://www.servergrove.com"><img src="../_static/servergrove.jpg" /></a> <br/><br/>
<form action="https://www.paypal.com/cgi-bin/webscr" method="post">
<input type="hidden" name="cmd" value="_s-xclick" />
<input type="hidden" name="hosted_button_id" value="BAE2E3XANQ77Y" />
<input type="Submit" value="Donate via PayPal" />
</form>
</div>
</div>
<div id="bot-rcnr">
<div class="tl"><!-- corner --></div>
</div>
</div>
<a class="githublink" href="http://github.com/doctrine">Fork me on GitHub</a>
</body>
</html>
|