<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:55:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20829] RUNNER_DEAD on document delete during update by _id or find by _id</title>
                <link>https://jira.mongodb.org/browse/SERVER-20829</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Given the following scenario:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;A document update begins for a {_id: 1234}&lt;/li&gt;
	&lt;li&gt;An _id entry is found for 1234 and the update yields to fetch the given document from disk&lt;/li&gt;
	&lt;li&gt;During the yield {_id: 1234} is deleted by another operation&lt;/li&gt;
	&lt;li&gt;The update operation fails as RUNNER_DEAD, returning an error&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The update should probably return successful with nupdated: 0 rather than error out. Looks like this is an issue for 2.6 only.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/v2.6/src/mongo/db/query/idhack_runner.cpp#L234-L236&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/v2.6/src/mongo/db/query/idhack_runner.cpp#L234-L236&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="233592">SERVER-20829</key>
            <summary>RUNNER_DEAD on document delete during update by _id or find by _id</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="david.storch@mongodb.com">David Storch</assignee>
                                    <reporter username="james.wahlin@mongodb.com">James Wahlin</reporter>
                        <labels>
                    </labels>
                <created>Thu, 8 Oct 2015 18:46:18 +0000</created>
                <updated>Tue, 22 Mar 2016 16:47:06 +0000</updated>
                            <resolved>Thu, 28 Jan 2016 14:49:40 +0000</resolved>
                                    <version>2.6.9</version>
                                    <fixVersion>2.6.12</fixVersion>
                                    <component>Querying</component>
                                        <votes>1</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1156607" author="xgen-internal-githook" created="Thu, 28 Jan 2016 14:48:34 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;dstorch&apos;, u&apos;name&apos;: u&apos;David Storch&apos;, u&apos;email&apos;: u&apos;david.storch@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20829&quot; title=&quot;RUNNER_DEAD on document delete during update by _id or find by _id&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20829&quot;&gt;&lt;del&gt;SERVER-20829&lt;/del&gt;&lt;/a&gt; return no results rather than RUNNER_DEAD from idhack on concurrent delete&lt;br/&gt;
Branch: v2.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b8a3d87501b15a881e135f45d4ef9de641b191a3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b8a3d87501b15a881e135f45d4ef9de641b191a3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1078209" author="rassi@10gen.com" created="Tue, 3 Nov 2015 16:28:08 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=james.wahlin&quot; class=&quot;user-hover&quot; rel=&quot;james.wahlin&quot;&gt;james.wahlin&lt;/a&gt;, I&apos;m tentatively assigning this ticket to you.  Please assign this back to me if you don&apos;t think you&apos;ll have time to get to this in the next week or two, and feel free to swing by the query team if you want to go over this in person.&lt;/p&gt;</comment>
                            <comment id="1068879" author="david.storch" created="Fri, 23 Oct 2015 13:15:56 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jon%40appboy.com&quot; class=&quot;user-hover&quot; rel=&quot;jon@appboy.com&quot;&gt;jon@appboy.com&lt;/a&gt;, Ramon is right that this discussion is best suited for other channels, but I thought I&apos;d still answer your outstanding question. The guarantees under concurrency offered by MongoDB 3.0 are the same regardless of the storage engine configuration. In general, queries against a mongod using WiredTiger can miss results. That said, I believe you are correct that this particular behavior of two contending updates by _id will not manifest on WiredTiger, since we will retry on write conflict.&lt;/p&gt;</comment>
                            <comment id="1068655" author="ramon.fernandez" created="Fri, 23 Oct 2015 02:06:23 +0000"  >&lt;p&gt;Thanks for your interest in MongoDB internals &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jon%40appboy.com&quot; class=&quot;user-hover&quot; rel=&quot;jon@appboy.com&quot;&gt;jon@appboy.com&lt;/a&gt;. This ticket is to get the RUNNER_DEAD error reported by James fixed in future versions of the server. For MongoDB-related discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;, where your questions will reach a larger audience. Questions like the ones above involving more discussion would be best posted on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt;. If you&apos;re interested in MongoDB development you can use the &lt;a href=&quot;http://groups.google.com/group/mongodb-dev&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-dev group&lt;/a&gt; instead. See also our &lt;a href=&quot;https://www.mongodb.org/about/support/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Technical Support page&lt;/a&gt; if you need additional support resources.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1064163" author="jonhyman" created="Mon, 19 Oct 2015 14:14:39 +0000"  >&lt;p&gt;Thanks, Dave.&lt;/p&gt;

&lt;p&gt;I&apos;m reading the section on isolation and note these two things, which are claimed to hold true for both MMAPv1 and WiredTiger:&lt;/p&gt;

&lt;p&gt;&amp;gt; Non-point-in-time read operations. Suppose a read operation begins at time t1 and starts reading documents. A write operation then commits an update to a document at some later time t2. The reader may see the updated version of the document, and therefore does not see a point-in-time snapshot of the data.&lt;br/&gt;
&amp;gt; Dropped results. Reads may miss matching documents that are updated or deleted during the course of the read operation. However, data that has not been modified during the operation will always be visible.&lt;/p&gt;

&lt;p&gt;In our situation, we have two processes that are doing something like this:&lt;/p&gt;

&lt;p&gt;db.my_collection.update({_id: X}, &lt;/p&gt;
{foo: &quot;bar&quot;}
&lt;p&gt;)&lt;br/&gt;
db.my_collection.update({_id: X}, &lt;/p&gt;
{baz: &quot;qux&quot;}
&lt;p&gt;)&lt;/p&gt;

&lt;p&gt;and it may be that case that one of these writes does not apply if update wins the race and causes the document to move; this is hinted at in the &quot;Dropped results&quot; section.&lt;/p&gt;

&lt;p&gt;Do you know if this would be the case in WiredTiger as well? My understand of WT is that due to its optimistic concurrency control, &quot;when the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation&quot; which I take to interpret as saying that both updates will get applied. Is it then just MMAPv1 that can fail to apply whichever write is the loser if a move happens?&lt;/p&gt;</comment>
                            <comment id="1064139" author="david.storch" created="Mon, 19 Oct 2015 13:59:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jon%40appboy.com&quot; class=&quot;user-hover&quot; rel=&quot;jon@appboy.com&quot;&gt;jon@appboy.com&lt;/a&gt;, in general queries may miss concurrently modified documents. For more details on the guarantees provided by the system under concurrency, please refer to the documentation: &lt;a href=&quot;http://docs.mongodb.org/manual/faq/concurrency/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/faq/concurrency/&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1063331" author="jonhyman" created="Fri, 16 Oct 2015 21:07:05 +0000"  >&lt;p&gt;Thanks. So if two threads make an update by id, but one causes a move, the other (if it loses the race) will fail to apply. In any highly concurrent system I would imagine this happens relatively frequently, as I&apos;m seeing in our case. Are you suggesting that the &quot;correct&quot;  and expected behavior is for one write to not apply? So we are to assume that any update by id (which in our case is the shard key) can return to the application and just not apply? Are we supposed to be confirming that all writes actually apply? Otherwise it seems like we&apos;re going to be prone to silent failures if the driver returns success. &lt;/p&gt;</comment>
                            <comment id="1062976" author="david.storch" created="Fri, 16 Oct 2015 16:15:04 +0000"  >&lt;p&gt;Hi Jon,&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;To clarify, in the case where a move happens, the write DOES occur, but Mongo just returns a failed error?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It depends on which write you mean, because there may be two writer threads. Say there is one thread doing a big update which causes an MMAP document move, and another thread which is doing a small update-by-id. If the first writer wins the race, is applied to the database, and causes the move to occur, the find-by-id write will &lt;em&gt;not&lt;/em&gt; occur. It is failing with RUNNER_DEAD on 2.6, but the expected behavior is that the update-by-id will report success and that it updated zero documents.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Should I retry the write or just ignore the error?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;This seems like an application-level decision. Drivers are not expected to retry the write in this case, but the application may want to do so.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Is this slated to be fixed in 2.6?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;This ticket is still marked as &quot;Needs Triage&quot;, so we have yet to decide whether to schedule this for a 2.6 release. Please watch the ticket for further updates on this.&lt;/p&gt;</comment>
                            <comment id="1062904" author="jonhyman" created="Fri, 16 Oct 2015 15:28:37 +0000"  >&lt;p&gt;Thanks for responding. To clarify, in the case where a move happens, the write DOES occur, but Mongo just returns a failed error? I&apos;m going to monkey patch the driver for this since we get it often (and it raises an unhandled exception for us). Should I retry the write or just ignore the error? Is this slated to be fixed in 2.6? We can&apos;t upgrade to 3.0 at the moment due to issues we&apos;re having with the official ruby driver that I&apos;m trying to figure out separately with that team.&lt;/p&gt;</comment>
                            <comment id="1062829" author="david.storch" created="Fri, 16 Oct 2015 14:49:27 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jon%40appboy.com&quot; class=&quot;user-hover&quot; rel=&quot;jon@appboy.com&quot;&gt;jon@appboy.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the feedback. After investigating the issue, it looks like this is a benign mistake in error reporting. The query engine is reporting an error on certain operations involving a find-by-_id when it should be reporting that the operation succeeded, but the document was not found.&lt;/p&gt;

&lt;p&gt;It is indeed the case that this can happen on when an update causes the document to be moved inside the MMAP storage engine. The expected behavior in this case is that the operation succeeds but does not find the document. The fix will be to eliminate these spuriously reported errors.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1062402" author="jonhyman" created="Fri, 16 Oct 2015 01:59:45 +0000"  >&lt;p&gt;In our highly concurrent environment, we&apos;re seeing this a couple of times per day after upgrading from 2.4.10 to 2.6.11. Actually, it may be slightly different because in our situation, my guess is that the document is moving.&lt;/p&gt;

&lt;p&gt;In this issue, Asya says that it could happen if the document moves. &lt;a href=&quot;https://groups.google.com/forum/#!searchin/mongodb-user/runner_dead/mongodb-user/dLgg8QBmUcY/RILrOjg8BQAJ&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/#!searchin/mongodb-user/runner_dead/mongodb-user/dLgg8QBmUcY/RILrOjg8BQAJ&lt;/a&gt; &lt;/p&gt;

&lt;p&gt;For errors it is happening to, I checked that the document was not deleted.&lt;/p&gt;</comment>
                            <comment id="1055790" author="david.storch" created="Thu, 8 Oct 2015 18:56:09 +0000"  >&lt;p&gt;Specifically the problem is that the IDHackRunner is marking itself as killed when the document it is trying to find by _id gets deleted. When this situation occurs on the 3.0 branch or on the master branch, we do not mark the query as killed. Instead, we make a copy of the document. The copy will get returned on the following call to getNext().&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.0.6/src/mongo/db/exec/idhack.cpp#L201-L207&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.0.6/src/mongo/db/exec/idhack.cpp#L201-L207&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We should probably do something similar inside the IDHackRunner on the 2.6 branch.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12451" key="com.atlassian.jira.plugin.system.customfieldtypes:multiversion">
                        <customfieldname>Backport Completed</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="15782">2.6.12</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 8 Oct 2015 18:56:09 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 2 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 2 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>james.wahlin@mongodb.com</customfieldvalue>
            <customfieldvalue>jonhyman</customfieldvalue>
            <customfieldvalue>rassi</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrksen:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsfcpr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="765">QuInt C (11/23/15)</customfieldvalue>
    <customfieldvalue id="859">Query F (02/01/16)</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsd4in:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>