<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:00:54 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-22607] Duplicate key error on replace with upsert</title>
                <link>https://jira.mongodb.org/browse/SERVER-22607</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We&apos;re getting a duplicate key error exception in a situation where one should never be occurring.  It occurs infrequently, but consistently.&lt;/p&gt;

&lt;p&gt;Our collection has a unique key defined on a combination of fields.  We then performa  `replaceOne()` with `upsert(true)`.  Generally this works fine, but a tiny percentage of the writes result in an error being thrown by mongo stating:&lt;/p&gt;

&lt;p&gt;    `com.mongodb.MongoWriteException: E11000 duplicate key error index`&lt;/p&gt;

&lt;p&gt;Our mongo instance is a single installation with no clustering and using the latest driver / server- 3.2.1.&lt;/p&gt;

&lt;p&gt;Our client is using the default connection pool and is sending a hundreds of requests per second to the database.  That said, our requests are unique in each is writing a different record to the database each time.  It&apos;s not the case where we&apos;d be sending a record with the same complex key via multiple threads at the same time.  &lt;/p&gt;

&lt;p&gt;It&apos;s worth noting that when this error occurs and an attempt made to re-upsert the exact same record record, it has succeeded every time.&lt;/p&gt;

&lt;p&gt;We&apos;re using the Java driver, but research has shown other people have run into this same error on StackOverflow:&lt;br/&gt;
&lt;a href=&quot;http://stackoverflow.com/questions/29305405/mongodb-impossible-e11000-duplicate-key-error-dup-key-when-upserting&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://stackoverflow.com/questions/29305405/mongodb-impossible-e11000-duplicate-key-error-dup-key-when-upserting&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="265288">SERVER-22607</key>
            <summary>Duplicate key error on replace with upsert</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="TheAndruu">Andrew Cuga [X]</reporter>
                        <labels>
                    </labels>
                <created>Sat, 13 Feb 2016 10:32:35 +0000</created>
                <updated>Wed, 18 May 2016 16:40:48 +0000</updated>
                            <resolved>Sat, 13 Feb 2016 13:45:14 +0000</resolved>
                                    <version>3.0.0</version>
                    <version>3.2.0</version>
                    <version>3.2.1</version>
                                                    <component>Write Ops</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="1267797" author="ramon.fernandez" created="Wed, 18 May 2016 16:40:48 +0000"  >&lt;p&gt;Apologies for the radio silence &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=TheAndruu&quot; class=&quot;user-hover&quot; rel=&quot;TheAndruu&quot;&gt;TheAndruu&lt;/a&gt;. This is to let you know that I&apos;ve opened &lt;a href=&quot;https://jira.mongodb.org/browse/DOCS-7904&quot; title=&quot;Enhance replaceOne() documentation on upsert behavior&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DOCS-7904&quot;&gt;&lt;del&gt;DOCS-7904&lt;/del&gt;&lt;/a&gt; to enhance the documentation as per your suggestion.&lt;/p&gt;

&lt;p&gt;Note also that the DOCS project is open to the public, so please feel free to open tickets there when you see areas of the documentation that can be improved.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1172731" author="theandruu" created="Sat, 13 Feb 2016 17:45:45 +0000"  >&lt;p&gt;Here&apos;s one example: the replaceOne upsert behavior documentation:&lt;br/&gt;
&lt;a href=&quot;https://docs.mongodb.org/v3.2/reference/method/db.collection.replaceOne/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.org/v3.2/reference/method/db.collection.replaceOne/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It mentions no warning as to the use of unique indices to ensure uniqueness.  This error is not occurring due to a race condition of the same record being written by two threads at the same time.  It&apos;s occurring when a record has already been persisted and propagated fully.&lt;/p&gt;

&lt;p&gt;Specifically from the documentation, this line is misleading as it&apos;s not a guarantee the operation will succeed despite this condition being met, with no warnings anywhere on the page:&lt;br/&gt;
&quot;Setting upsert: true would insert the document if no match was found. See Replace with Upsert&quot;&lt;/p&gt;</comment>
                            <comment id="1172721" author="ramon.fernandez" created="Sat, 13 Feb 2016 17:10:14 +0000"  >&lt;p&gt;Digging through the documentation I can see that &lt;a href=&quot;https://docs.mongodb.org/v2.4/reference/method/db.collection.update/#upsert-behavior&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this behavior has been documented&lt;/a&gt; since 2013 (&lt;a href=&quot;https://github.com/mongodb/docs/commit/5fff9af89066b106ab31bdfe7f1f6cf2919f8134&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;see this commit&lt;/a&gt;): &lt;tt&gt;update()&lt;/tt&gt; with &lt;tt&gt;upsert&lt;/tt&gt; set to &lt;tt&gt;true&lt;/tt&gt; may insert duplicate documents unless one uses unique indexes. The same behavior applies to &lt;a href=&quot;https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/#db-collection-findandmodify&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;findAndModify()&lt;/tt&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;That being said, if you&apos;ve found places in our documentation that you think may lead users to believe that certain operations are atomic when in fact they&apos;re not, could you please link them here? I definitely see value in reviewing and improving them for other users.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1172694" author="theandruu" created="Sat, 13 Feb 2016 16:25:04 +0000"  >&lt;p&gt;Understandable if there&apos;s already a ticket open on this issue.  &lt;/p&gt;

&lt;p&gt;Thought it&apos;s unfair to say this behavior is expected.  According to the spec, this condition should never occur.  Luckily we were overly cautious and used unique indices as a additional layer of protection.  If not, we&apos;d be running into serious issues as a result of mongo behaving contrary to expectations.&lt;/p&gt;

&lt;p&gt;There are at least 12 issues recorded in JIRA on this issue, dating to July 2013.  Is there a chance this will either get fixed or at least documented better to explain it&apos;s not an atomic operation?   It&apos;s a serious flaw to allow the upsert operation appear as atomic when it in fact would be inserting duplicate records if not for a unique index.&lt;/p&gt;</comment>
                            <comment id="1172639" author="ramon.fernandez" created="Sat, 13 Feb 2016 13:44:34 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=TheAndruu&quot; class=&quot;user-hover&quot; rel=&quot;TheAndruu&quot;&gt;TheAndruu&lt;/a&gt;, the behavior you&apos;re seeing is expected, and your solution (retrying at the application level) is the correct one. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14322&quot; title=&quot;Retry on predicate unique index violations of update + upsert -&amp;gt; insert when possible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14322&quot;&gt;&lt;del&gt;SERVER-14322&lt;/del&gt;&lt;/a&gt; is opened to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14322?focusedCommentId=1130310&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1130310&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;discuss whether the server should do the retry&lt;/a&gt;, so feel free to watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14322&quot; title=&quot;Retry on predicate unique index violations of update + upsert -&amp;gt; insert when possible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14322&quot;&gt;&lt;del&gt;SERVER-14322&lt;/del&gt;&lt;/a&gt; for updates and/or vote for it.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="143209">SERVER-14322</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="287561">DOCS-7904</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sat, 13 Feb 2016 13:44:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 39 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 39 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>TheAndruu</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkh1b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsidin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;// Unique index on the collection&lt;br/&gt;
Document index = new Document(&quot;brickId&quot;, 1);&lt;br/&gt;
index.append(&quot;date&quot;, 1);&lt;br/&gt;
index.append(&quot;brickCategory&quot;, 1);&lt;br/&gt;
index.append(&quot;mortarCategory&quot;, 1);&lt;br/&gt;
collection.createIndex(index, new IndexOptions().unique(true));&lt;/p&gt;

&lt;p&gt;// Document locating the record to replace&lt;br/&gt;
Document searchDoc = new Document(&quot;brickId&quot;, &quot;569f6ffa8ddccfb6412d9dc2&quot;);&lt;br/&gt;
searchDoc.append(&quot;date&quot;, new Date(1296432000000));&lt;br/&gt;
searchDoc.append(&quot;brickCategory&quot;, &quot;Red&quot;);&lt;br/&gt;
searchDoc.append(&quot;mortarCategory&quot;, &quot;Steel&quot;);&lt;br/&gt;
return searchDoc;&lt;/p&gt;


&lt;p&gt;// Document being inserted (or replaced)&lt;br/&gt;
Document doc = new Document();&lt;br/&gt;
doc.append(brickId, &quot;569f6ffa8ddccfb6412d9dc2&quot;);&lt;br/&gt;
doc.append(&quot;brickCategory&quot;, &quot;Red&quot;);&lt;br/&gt;
doc.append(&quot;mortarCategory&quot;, &quot;Steel&quot;);&lt;br/&gt;
doc.append(&quot;amount, 100);&lt;br/&gt;
doc.append(&quot;date&quot;, new Date());&lt;/p&gt;


&lt;p&gt;// Code used to do replace with upsert(true) call&lt;br/&gt;
UpdateResult result = getCollection().replaceOne(searchDoc, doc, new UpdateOptions().upsert(true));&lt;/p&gt;


&lt;p&gt;// Error bubbling up from mongo&lt;br/&gt;
com.mongodb.MongoWriteException: E11000 duplicate key error index: mydb.Costs.$brickId_1_date_1_brickCategory_1_mortarCategory_1 dup key: { : &quot;569f6ffa8ddccfb6412d9dc2&quot;, : new Date(1296432000000), : &quot;Red&quot;, : &quot;Steel&quot; }&lt;br/&gt;
	at com.mongodb.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:523) ~&lt;span class=&quot;error&quot;&gt;&amp;#91;MongoCollectionImpl.class:na&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.mongodb.MongoCollectionImpl.replaceOne(MongoCollectionImpl.java:344) ~&lt;span class=&quot;error&quot;&gt;&amp;#91;MongoCollectionImpl.class:na&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Other steps to reproduce:&lt;br/&gt;
&lt;a href=&quot;http://stackoverflow.com/questions/29305405/mongodb-impossible-e11000-duplicate-key-error-dup-key-when-upserting&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://stackoverflow.com/questions/29305405/mongodb-impossible-e11000-duplicate-key-error-dup-key-when-upserting&lt;/a&gt;&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsbyy7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>