<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:19:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49161] Change stream event with empty fullDocument field for existing document</title>
                <link>https://jira.mongodb.org/browse/SERVER-49161</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;On investigation flaky test in our codebase I found strange issue: sometimes I got change stream&#160;&lt;tt&gt;update&lt;/tt&gt;&#160;event with empty&#160;&lt;tt&gt;fullDocument&lt;/tt&gt;&#160;field for existing document.&lt;/p&gt;

&lt;p&gt;I reproduced this behaviour by looping with same collection:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;drop collection;&lt;/li&gt;
	&lt;li&gt;shard collection;&lt;/li&gt;
	&lt;li&gt;open change stream;&lt;/li&gt;
	&lt;li&gt;run background thread which add ~2000 records and then update ~100 records;&lt;/li&gt;
	&lt;li&gt;read change stream and check then update event has non-empty&#160;&lt;tt&gt;fullDocument&lt;/tt&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Environment details:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;one shard from single-node replicaset;&lt;/li&gt;
	&lt;li&gt;single-node configuration replicaset;&lt;/li&gt;
	&lt;li&gt;3 mongos.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;On my computer, a spike of errors occurs in the first 10-30 iterations. Then the probability of errors greatly decreases.&lt;/p&gt;

&lt;p&gt;I wrote a simple program with Golang for reproducing this issue (&lt;tt&gt;main.go&lt;/tt&gt;). This program uses&#160;&lt;tt&gt;go.mongodb.org/mongo-driver v1.3.4&lt;/tt&gt;&#160;for MongoDB access but originally this issue was reproduced with&#160;&lt;tt&gt;github.com/globalsign/mgo&lt;/tt&gt;&#160;fork.&lt;/p&gt;</description>
                <environment>I tested on host:&lt;br/&gt;
&amp;nbsp;- OS: Ubuntu 20.04 LTS&lt;br/&gt;
&amp;nbsp;- CPU: Intel i7-8700K&lt;br/&gt;
&amp;nbsp;- RAM: 64Gb&lt;br/&gt;
&amp;nbsp;- SSD: Samsung NVMe 500Gb</environment>
        <key id="1394401">SERVER-49161</key>
            <summary>Change stream event with empty fullDocument field for existing document</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="rishab.joshi@mongodb.com">Rishab Joshi</assignee>
                                    <reporter username="bozaro@gmail.com">Artem Navrotskiy</reporter>
                        <labels>
                    </labels>
                <created>Mon, 29 Jun 2020 12:56:48 +0000</created>
                <updated>Mon, 11 Oct 2021 12:41:15 +0000</updated>
                            <resolved>Tue, 5 Oct 2021 10:55:04 +0000</resolved>
                                    <version>4.0.19</version>
                    <version>4.2.8</version>
                    <version>5.0.0</version>
                    <version>4.4.7</version>
                                                    <component>Change streams</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="4101841" author="JIRAUSER1257558" created="Tue, 5 Oct 2021 10:55:05 +0000"  >&lt;p&gt;I ran the scenarios multiple times with/without change stream optimization feature flag. I can see that everytime with feature flag disabled the issue happens and with the feature flag enabled the issue does not happen. It looks like the post-image stage reordering work fixed the issue. I can&apos;t see this issue happening now and hence marking it as a duplicate of the post image work done as part of the change stream optimization project.&lt;/p&gt;</comment>
                            <comment id="3943633" author="tommaso.tocci" created="Thu, 15 Jul 2021 14:52:08 +0000"  >&lt;p&gt;I&apos;ve been able to reproduce the problem using this js test: &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/325350/325350_change_streams_empty_document.js&quot; title=&quot;change_streams_empty_document.js attached to SERVER-49161&quot;&gt;change_streams_empty_document.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; you can run it through resmoke by using:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;resmoke.py --mongosSetParameters=&quot;{logComponentVerbosity: {sharding: 3}}&quot; --suites=sharding &amp;lt;test&amp;gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;From what I understood this is what is happening:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;mongosB has information about collection &quot;db.myColl&quot; with UUID_1&lt;/li&gt;
	&lt;li&gt;&quot;db.myColl&quot; gets dropped and recreated through mongosA and the new collection has UUID_2&lt;/li&gt;
	&lt;li&gt;a ChangeStream with &quot;fullDocument&quot; option is opened for &quot;db.myColl&quot; on mongosA (that has still stale metadata for the collection)&lt;/li&gt;
	&lt;li&gt;An update event arrives to mongosB through the opened ChangeStream&lt;/li&gt;
	&lt;li&gt;mognosB &lt;a href=&quot;https://github.com/mongodb/mongo/blob/331759573b9e60847a605e4d09292293e7cfa472/src/mongo/db/pipeline/document_source_change_stream_lookup_post_image.cpp#L133-L134&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;calls &quot;MongosProcessInterface::&lt;/a&gt;[lookupSingleDocument&quot; passing the UUID_2 contained in the resume token of the change stream event|https://github.com/mongodb/mongo/blob/331759573b9e60847a605e4d09292293e7cfa472/src/mongo/db/pipeline/document_source_change_stream_lookup_post_image.cpp#L133-L134].&lt;/li&gt;
	&lt;li&gt;At this point mongosB will realize that the UUID_2 &lt;a href=&quot;https://github.com/mongodb/mongo/blob/331759573b9e60847a605e4d09292293e7cfa472/src/mongo/db/pipeline/process_interface/mongos_process_interface.cpp#L72-L76&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;received with the event don&apos;t match the one on the cached routing table &lt;/a&gt;(UUID_1) and it will through&#160; a NamepsaceNotFound exception&lt;/li&gt;
	&lt;li&gt;The exception will be &lt;a href=&quot;https://github.com/mongodb/mongo/blob/331759573b9e60847a605e4d09292293e7cfa472/src/mongo/db/pipeline/process_interface/mongos_process_interface.cpp#L214-L218&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;catched by the lookupSingleDocument&lt;/a&gt; function that will simply return an empty &quot;fullDocument&quot;.&lt;/li&gt;
&lt;/ol&gt;



&lt;p&gt;In general the information that we get through the ChangeStream could be misaligned with respect to the metadata that we have cached on the mongos. In fact we could have:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;ChangeStream behind local metadata: mongos receives events for an old collection that have been already dropped and for which we don&apos;t have any information cached anymore.&lt;/li&gt;
	&lt;li&gt;ChangeStream ahead of local metadata: mongos receives events for a new collection for which we don&apos;t still have cached information&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The problem described in this ticket is a symptom of the latter.&lt;/p&gt;

&lt;p&gt;Unfortunately by just comparing the received UUID with the cached one we can&apos;t differentiate between the two situations, because we don&apos;t know which of the two UUID is newer. Recently we added a timestamp field to the collection entry that can be used to properly compare collection&apos;s metadata.&lt;/p&gt;

&lt;p&gt;I&apos;m not sure if we can use the oplog entry timestamp to campare it with the collection metadata one, otherwise another solution solution&#160; would be to add the collection timestamp along with the UUID in the ShangeStream events.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1652965">SERVER-55309</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1652965">SERVER-55309</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="325350" name="change_streams_empty_document.js" size="3470" author="tommaso.tocci@mongodb.com" created="Thu, 15 Jul 2021 14:08:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 7 Jul 2021 18:00:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 18 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-55309'>SERVER-55309</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>rishab.joshi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 18 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bozaro@gmail.com</customfieldvalue>
            <customfieldvalue>rishab.joshi@mongodb.com</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxsa2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2yev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3620">Query 2020-11-30</customfieldvalue>
    <customfieldvalue id="3621">Query 2020-12-14</customfieldvalue>
    <customfieldvalue id="3622">Query 2020-12-28</customfieldvalue>
    <customfieldvalue id="4459">Query 2021-01-11</customfieldvalue>
    <customfieldvalue id="4460">Query 2021-01-25</customfieldvalue>
    <customfieldvalue id="4462">Query Execution 2021-02-22</customfieldvalue>
    <customfieldvalue id="4464">Query Execution 2021-03-08</customfieldvalue>
    <customfieldvalue id="4466">Query Execution 2021-03-22</customfieldvalue>
    <customfieldvalue id="4468">Query Execution 2021-04-05</customfieldvalue>
    <customfieldvalue id="4470">Query Execution 2021-04-19</customfieldvalue>
    <customfieldvalue id="4980">Sharding EMEA 2021-07-12</customfieldvalue>
    <customfieldvalue id="5267">QE 2021-10-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;On Linux host:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;checkout&#160;&lt;a href=&quot;https://github.com/bozaro/change-stream-test/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/bozaro/change-stream-test/&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;run `./run.sh`&lt;/li&gt;
&lt;/ul&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxrwbr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>