<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:30:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-75607] Change Stream: Shard keys not captured in documentKey on delete operations performed directly on mongo instances</title>
                <link>https://jira.mongodb.org/browse/SERVER-75607</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Change Stream: Shard keys are captured in &lt;tt&gt;documentKey&lt;/tt&gt; on delete operations performed when connected to &lt;tt&gt;mongos&lt;/tt&gt; and &lt;tt&gt;mongodb+srv&lt;/tt&gt;, but not directly to MongoDB instances.&lt;/p&gt;

&lt;p&gt;Perform the following operations listed in the &lt;tt&gt;Steps To Reproduce&lt;/tt&gt; section while reading from ChangeStream.&lt;/p&gt;

&lt;p&gt;It can be seen that shard keys and their values are captured by the change stream when the &lt;tt&gt;delete&lt;/tt&gt; operations were performed by &lt;tt&gt;mongosh&lt;/tt&gt; connections that were connected to &lt;tt&gt;mongos&lt;/tt&gt; or &lt;tt&gt;mongodb+srv&lt;/tt&gt; (Cases 1 &amp;amp; 2), but not by connections that were directly connected to the MongoDB database instances (Case 3). This behavior is exhibited only for &lt;tt&gt;delete&lt;/tt&gt; operations. It was noted that &lt;tt&gt;update&lt;/tt&gt; change events had shard keys in their document keys in all these scenarios.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2305179">SERVER-75607</key>
            <summary>Change Stream: Shard keys not captured in documentKey on delete operations performed directly on mongo instances</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="sebastien.mendez@mongodb.com">Sebastien Mendez</assignee>
                                    <reporter username="ajay2589@gmail.com">Ajay Mathias</reporter>
                        <labels>
                            <label>changestreams</label>
                    </labels>
                <created>Mon, 3 Apr 2023 17:07:09 +0000</created>
                <updated>Wed, 15 Nov 2023 17:19:28 +0000</updated>
                            <resolved>Wed, 15 Nov 2023 17:19:27 +0000</resolved>
                                    <version>6.0.4</version>
                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="5882616" author="JIRAUSER1260027" created="Wed, 15 Nov 2023 17:19:28 +0000"  >&lt;p&gt;Based on my tests and my investigation, the shardKey is never added to the documentKey when the commands are directly sent to the shards, and this is true for deletes but also for inserts and updates.&lt;/p&gt;

&lt;p&gt;Therefore, this is consistent and will not be modified as it is crucial not to bypass mongos and send commands directly to mongod: they will not be routed to the correct shard, resulting in &quot;orphaned&quot; documents and document not effectively deleted, and thus introducing data inconsistencies.&lt;/p&gt;</comment>
                            <comment id="5740540" author="JIRAUSER1270641" created="Fri, 29 Sep 2023 13:11:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ajay2589%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;ajay2589@gmail.com&quot;&gt;ajay2589@gmail.com&lt;/a&gt;- Could you please add some context around connecting to a shard directly (mongosh &quot;mongodb://xxxx-shard-00-00.yyyy.mongodb.net:27017,xxxx-shard-00-01.yyyy.mongodb.net:27017,xxxx-shard-00-02.yyyy.mongodb.net:27017) and conducting a delete?  &lt;/p&gt;

&lt;p&gt;I would like to understand is this your typical workflow or were you testing and found the behavior difference. Thanks!&lt;/p&gt;</comment>
                            <comment id="5738055" author="JIRAUSER1260027" created="Thu, 28 Sep 2023 15:56:03 +0000"  >&lt;p&gt;If you attempt to insert a document into a specific shard that should belong to another shard, MongoDB will not automatically route the document to the correct shard. Instead, it will be stored on the shard where you attempted the insertion, effectively becoming an &quot;orphan&quot; document.&lt;br/&gt;
These orphaned documents will not be visible through the mongos router, and may be deleted by a background job.&lt;br/&gt;
The same apply for delete, as the document from the correct shard won&apos;t be deleted neither.&lt;/p&gt;

&lt;p&gt;To ensure proper data distribution, i&lt;b&gt;t&apos;s crucial to insert/update/delete documents using the mongos router&lt;/b&gt;, which will correctly route the command to the appropriate shard.&lt;br/&gt;
Note that documents that are not owned by that shard but are affected by the update/insert/delete do not generate any changestream events.&lt;br/&gt;
Attempting to act on documents directly into a specific shard is not a recommended practice in a sharded MongoDB cluster, as it can lead to data inconsistencies.&lt;/p&gt;

&lt;p&gt;Could you please clarify your requirement for performing document insertion/updating/deletion within a sharded collection directly on the individual shards?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2312318">SERVER-75993</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 30 Jun 2023 13:52:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>sebastien.mendez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>ajay2589@gmail.com</customfieldvalue>
            <customfieldvalue>matt.panton@mongodb.com</customfieldvalue>
            <customfieldvalue>sebastien.mendez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i232wn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1id24:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7249">QE 2023-09-18</customfieldvalue>
    <customfieldvalue id="7251">QE 2023-10-02</customfieldvalue>
    <customfieldvalue id="7254">QE 2023-10-16</customfieldvalue>
    <customfieldvalue id="7255">QE 2023-10-30</customfieldvalue>
    <customfieldvalue id="7257">QE 2023-11-13</customfieldvalue>
    <customfieldvalue id="7259">QE 2023-11-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;&lt;b&gt;Case 1&lt;/b&gt;: Connect using &lt;tt&gt;mongodb+srv&lt;/tt&gt; URL&lt;br/&gt;
&lt;tt&gt;mongosh &quot;mongodb+srv://xxxx.yyyy.mongodb.net&quot; &#160;&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Perform delete operation on a collection with shard key &quot;name&quot;&lt;/p&gt;

&lt;p&gt;ChangeStream captured has document key - &lt;tt&gt;documentKey={&quot;name&quot;: &quot;1&quot;, &quot;_id&quot;: 1&lt;/tt&gt;}&lt;/p&gt;

&lt;p&gt;Both &lt;tt&gt;_id&lt;/tt&gt; and shard key value are present.&lt;/p&gt;


&lt;p&gt;&lt;b&gt;Case 2&lt;/b&gt;: Connect to mongos&lt;br/&gt;
&lt;tt&gt;mongosh &quot;mongodb://xxxx-shard-00-00.yyyy.mongodb.net:27016,xxxx-shard-00-01.yyyy.mongodb.net:27016,xxxx-shard-00-02.yyyy.mongodb.net:27016&quot; --tls&lt;/tt&gt;&lt;br/&gt;
Perform delete operation on a collection with shard key &quot;name&quot;&lt;/p&gt;

&lt;p&gt;ChangeStream captured has document key - &lt;tt&gt;documentKey={&quot;name&quot;: &quot;6&quot;, &quot;_id&quot;: 6&lt;/tt&gt;}&lt;/p&gt;

&lt;p&gt;Both &lt;tt&gt;_id&lt;/tt&gt; and shard key value are present.&lt;/p&gt;


&lt;p&gt;&lt;b&gt;Case 3&lt;/b&gt;: Connect to mongod instances&lt;br/&gt;
&lt;tt&gt;mongosh &quot;mongodb://xxxx-shard-00-00.yyyy.mongodb.net:27017,xxxx-shard-00-01.yyyy.mongodb.net:27017,xxxx-shard-00-02.yyyy.mongodb.net:27017&quot; --tls&lt;/tt&gt;&lt;br/&gt;
Perform delete operation on a collection with shard key &quot;name&quot;&lt;/p&gt;

&lt;p&gt;ChangeStream captured has document key - &lt;tt&gt;documentKey={&quot;_id&quot;: 2&lt;/tt&gt;}&lt;/p&gt;

&lt;p&gt;Only _id is present, although the document had a non-empty shard key value.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i22p1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>