<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:47:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18304] Duplicate &quot;value&quot; fields in the findAndModify command response</title>
                <link>https://jira.mongodb.org/browse/SERVER-18304</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #EEEEEE;border-color: #ccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #ccc;background-color: #6CB33F;&quot;&gt;&lt;b&gt;Issue Status as of Sep 29, 2015&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #EEEEEE;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;br/&gt;
On MongoDB instances running with document-level concurrency storage engines, concurrent modifications of the same document by multiple clients may cause &lt;tt&gt;WriteConflictExceptions&lt;/tt&gt; to be thrown. These write conflicts are handled internally by retrying the operation.&lt;/p&gt;

&lt;p&gt;A bug that affects both &lt;tt&gt;findAndModify&lt;/tt&gt; remove operations and &lt;tt&gt;findAndModify&lt;/tt&gt; update operations with &lt;tt&gt;new=false&lt;/tt&gt; may cause the &quot;value&quot; field to be included in the server&apos;s response multiple times. Each occurrence of the field is likely to be associated with different documents or different versions of the same document.&lt;/p&gt;

&lt;p&gt;The server&apos;s response was mutated to include the old version of the document prior to the delete or update operation taking place. If faced with a write conflict, the operation would be retried on a potentially different document. Each attempt to perform the operation would cause that document to be appended to the server&apos;s response under a new &quot;value&quot; field. This means that the last &quot;value&quot; field present in the server&apos;s response is the actual document that was updated or removed by the client.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;br/&gt;
How a driver handles duplicate fields in the server&apos;s response isn&apos;t standardized behavior. The most common scenarios are that:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;The driver may choose the first &quot;value&quot; field in the response. This corresponds to the document that was first attempted to be updated or deleted, but wasn&apos;t actually the one modified by this client.&lt;/li&gt;
	&lt;li&gt;The driver may choose the last &quot;value&quot; field in the response. This corresponds to the document that was updated or deleted by this client.&lt;/li&gt;
	&lt;li&gt;The driver may return an error because the &quot;value&quot; field was present in the response multiple times.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;For sufficiently large documents with many concurrent &lt;tt&gt;findAndModify&lt;/tt&gt; operations, it is possible to trigger enough write conflicts such that the server&apos;s response exceeeds 16MB. This causes an error to be returned to the client saying that the BSONObj size is invalid.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WORKAROUNDS&lt;/b&gt;&lt;br/&gt;
None.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;AFFECTED VERSIONS&lt;/b&gt;&lt;br/&gt;
The 3.0 release series up until (and including) 3.0.2 are affected by this issue. Only MongoDB instances running with document-level concurrency storage engines (e.g. WiredTiger and RocksDB) are affected. MMAPv1 doesn&apos;t have a notion of write conflicts and is &lt;b&gt;not&lt;/b&gt; affected.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FIX VERSION&lt;/b&gt;&lt;br/&gt;
The fix is included in the 3.0.3 production release.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;h6&gt;&lt;a name=&quot;Originaldescription&quot;&gt;&lt;/a&gt;Original description&lt;/h6&gt;
&lt;p&gt;I&apos;m using a mongo collection as a queue, on mongo v3.0.2 with wiredtiger; producers use simple inserts and consumers use findAndModify with a sort descending on a timestamp, and remove option&lt;/p&gt;

&lt;p&gt;I&apos;ve noticed that when I use one producer and multiple consumers in separated threads, in quite a many cases more than one thread return or obtain the same element out of the findAndModify. I&apos;ve reread the docs on the semantics, and this seems a bug to me&lt;/p&gt;</description>
                <environment></environment>
        <key id="201691">SERVER-18304</key>
            <summary>Duplicate &quot;value&quot; fields in the findAndModify command response</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="max.hirschhorn@mongodb.com">Max Hirschhorn</assignee>
                                    <reporter username="pep.martinez">Pep Martinez</reporter>
                        <labels>
                    </labels>
                <created>Sun, 3 May 2015 08:30:24 +0000</created>
                <updated>Mon, 5 Oct 2015 20:16:10 +0000</updated>
                            <resolved>Mon, 4 May 2015 23:09:04 +0000</resolved>
                                    <version>3.0.2</version>
                                    <fixVersion>3.0.3</fixVersion>
                                    <component>Concurrency</component>
                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="904846" author="xgen-internal-githook" created="Mon, 4 May 2015 23:07:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;visemet&apos;, u&apos;name&apos;: u&apos;Max Hirschhorn&apos;, u&apos;email&apos;: u&apos;max.hirschhorn@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18304&quot; title=&quot;Duplicate &amp;quot;value&amp;quot; fields in the findAndModify command response&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18304&quot;&gt;&lt;del&gt;SERVER-18304&lt;/del&gt;&lt;/a&gt; Add findAndModify FSM workloads as regression tests.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/50922d9f626758855326f6b5a06e940269189e11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/50922d9f626758855326f6b5a06e940269189e11&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="904760" author="xgen-internal-githook" created="Mon, 4 May 2015 21:49:04 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;visemet&apos;, u&apos;name&apos;: u&apos;Max Hirschhorn&apos;, u&apos;email&apos;: u&apos;max.hirschhorn@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18304&quot; title=&quot;Duplicate &amp;quot;value&amp;quot; fields in the findAndModify command response&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18304&quot;&gt;&lt;del&gt;SERVER-18304&lt;/del&gt;&lt;/a&gt; Only call _appendHelper() if update/remove succeeds.&lt;/p&gt;

&lt;p&gt;Otherwise the response will contain the &quot;value&quot; field multiple times&lt;br/&gt;
because the command result is mutated even when the operation does&lt;br/&gt;
not succeed (due to a WriteConflictException). Depending on the client&lt;br/&gt;
logic, e.g. using the first occurrence, it may end up claiming to have&lt;br/&gt;
deleted a document that was actually deleted by another client.&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/feef878308e0ee990a17826073d8e4db9ec1657b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/feef878308e0ee990a17826073d8e4db9ec1657b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="904242" author="max.hirschhorn@10gen.com" created="Mon, 4 May 2015 15:02:31 +0000"  >&lt;p&gt;I was able to reproduce this with an FSM workload (attached) that repeatedly removes documents from a collection using the findAndModify command. Note that a similar issue exists for updates with new=false.&lt;/p&gt;

&lt;p&gt;While prototyping the workload, I found that the collection was still empty at the end of the workload. That is to say, despite some threads claiming to have removed the same document and running for a fixed number of iterations, all documents had actually been removed.&lt;/p&gt;

&lt;p&gt;From reading through the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.0.2/src/mongo/db/commands/find_and_modify.cpp#L371&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;CmdFindAndModify implementation in 3.0.2&lt;/a&gt;, I realized that we would have already filled in the values for the result object prior to calling &lt;tt&gt;deleteObjects()&lt;/tt&gt;, which may trigger a WriteConflictException. By moving the &lt;tt&gt;_appendHelper()&lt;/tt&gt; call to after the &lt;tt&gt;if (found)&lt;/tt&gt; statement, I can no longer reproduce the issue. (Note that the actual fix will be slightly more involved because we&apos;ll need to store a copy of the removed document for mmapv1.)&lt;/p&gt;

&lt;p&gt;To summarize, each thread is still removing a distinct document, but the response that comes back from the server has the wrong value for what document was removed.&lt;/p&gt;</comment>
                            <comment id="904085" author="pep.martinez" created="Mon, 4 May 2015 10:37:03 +0000"  >&lt;p&gt;I&apos;ve also noticed that v3-mmapv1 and v2.6 are not affected by this issue&lt;/p&gt;</comment>
                            <comment id="904069" author="ramon.fernandez" created="Mon, 4 May 2015 09:08:43 +0000"  >&lt;p&gt;Thanks for the report &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=pep.martinez&quot; class=&quot;user-hover&quot; rel=&quot;pep.martinez&quot;&gt;pep.martinez&lt;/a&gt;, we&apos;re able to observe the behavior you describe and we&apos;re investigating.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="194328">SERVER-17902</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="72743" name="findAndModify_remove_queue.js" size="5128" author="max.hirschhorn@mongodb.com" created="Mon, 4 May 2015 15:02:31 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 3 May 2015 19:26:13 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 41 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WRITING-1428'>WRITING-1428</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 41 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>pep.martinez</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjxrr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs8zc7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="490">Quint Iteration 3</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;my code involves quite a lot of details and it&apos;s c++; I&apos;m still to build a simple test case in, for example, python&lt;/p&gt;

&lt;p&gt;In short: &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;one producer, running inserts like db.coll.insert (
{d: data, t: now()}
&lt;p&gt;). Producer in a tight loop inserting 100K elements&lt;/p&gt;&lt;/li&gt;
	&lt;li&gt;5-10 consumers, one in each thread, getting elements with&lt;br/&gt;
db.coll.findAndModify({&lt;br/&gt;
    sort: 
{ t: 1 }
&lt;p&gt;,&lt;br/&gt;
    remove: true&lt;br/&gt;
})&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;consumers end up returning about 140-160K elements. Some are duplicated even thrice&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11858"><![CDATA[Completed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsb45z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>