<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:40:48 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35740] Report high-water-mark resume token with each (possibly empty) change stream batch</title>
                <link>https://jira.mongodb.org/browse/SERVER-35740</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Original title: Change streams with no results cannot be resumed once the oplog has rolled over&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Suppose you want to open a change stream to listen for an event that happens relatively rarely, maybe once a week or so. The &apos;resumeAfter&apos; protocol implemented with drivers isn&apos;t very helpful here because you will never get a resume token until that rare event happens. Thus, if there&apos;s a network error of some sort and your stream has to be resumed, you have no option but to start a new one. But starting a new one might start ahead of where the last one left off!&lt;/p&gt;

&lt;p&gt;Starting in 4.0, the desired protocol for opening a change stream with a driver is to start watching at an operation time, any operation time from around the time the stream is opened will work for the examples here. This means that if you want to resume your stream before you see any resume token, you can just remember the time you started and start again from that time. This is going to scan some extra data, but at least you won&apos;t miss anything.&lt;/p&gt;

&lt;p&gt;However, there&apos;s still a problem with very infrequent events. If enough time passes between opening the stream and the interesting event, the operation time you started with will no longer be present in the oplog. Supposing this is the case and there is a network error, the driver will attempt to re-open the stream and the server will return an error because we cannot start a change stream that far in the past (we would be missing events).&lt;/p&gt;

&lt;p&gt;To fix this, we could do one of the following:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Augment the driver protocol to expose some of the information we use internally between mongos and mongod to show progress despite the lack of events. See &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29929&quot; title=&quot;Report highest observed cluster time from change stream on each shard&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29929&quot;&gt;&lt;del&gt;SERVER-29929&lt;/del&gt;&lt;/a&gt;. Then the drivers could use this time to resume and simultaneously avoid re-scanning a ton on resume and avoid concerns about rolling off the oplog.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Relax the assertions in the server to allow using an operation time that&apos;s no longer in the oplog.jj&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="562934">SERVER-35740</key>
            <summary>Report high-water-mark resume token with each (possibly empty) change stream batch</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="bernard.gorman@mongodb.com">Bernard Gorman</assignee>
                                    <reporter username="charlie.swanson@mongodb.com">Charlie Swanson</reporter>
                        <labels>
                    </labels>
                <created>Fri, 22 Jun 2018 12:18:23 +0000</created>
                <updated>Sun, 29 Oct 2023 22:30:31 +0000</updated>
                            <resolved>Fri, 1 Mar 2019 15:06:03 +0000</resolved>
                                                    <fixVersion>4.0.7</fixVersion>
                    <fixVersion>4.1.9</fixVersion>
                                    <component>Aggregation Framework</component>
                                        <votes>2</votes>
                                    <watches>19</watches>
                                                                                                                <comments>
                            <comment id="2063488" author="jonathan.green" created="Thu, 15 Nov 2018 17:48:06 +0000"  >&lt;p&gt;Hi Jack,&lt;/p&gt;

&lt;p&gt;*I asked the team working directly on this for a timeline, here&apos;s their&lt;br/&gt;
response:*&lt;br/&gt;
Hello! We are actively working on this one, yes. The timeline is a bit hard&lt;br/&gt;
to predict because this one involves first doing the changes on the server&lt;br/&gt;
in the master branch, then seeing if/how we can backport some to the 4.0&lt;br/&gt;
release, &lt;em&gt;then&lt;/em&gt; implementing new driver APIs on top of that, and finally&lt;br/&gt;
releasing a new version of 4.0 and of whichever driver the customer is&lt;br/&gt;
using. I would say absolute best case would be a month and a half or two&lt;br/&gt;
months, but more likely lagging behind that - we&apos;re still in the early&lt;br/&gt;
phases so it&apos;s hard to anticipate whether we&apos;ll encounter complexities.&lt;/p&gt;

&lt;p&gt;*I followed up by asking if this is an issue that you may potentially see&lt;br/&gt;
in development and that you would not see in production. Here&apos;s their&lt;br/&gt;
response:*&lt;br/&gt;
Right. We expect this to only be an issue if the events are far apart&lt;/p&gt;

&lt;p&gt;Because this issue only arises when there is a long oplog between triggers,&lt;br/&gt;
you may not see the issue in production. *Would you consider testing under&lt;br/&gt;
a production-like environment?*&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Jonathan&lt;/p&gt;


&lt;p&gt;On Thu, Nov 15, 2018 at 10:28 AM, DORDONNE Jacques-olivier (Jira) &amp;lt;&lt;/p&gt;




&lt;p&gt;&amp;#8211; &lt;/p&gt;


{ name     : &quot;Jonathan Green&quot;,

  phone    : &quot;866-237-8815 ext.4027&quot;,

  location : &quot;Austin, TX&quot;,}</comment>
                            <comment id="2063330" author="jodordonne" created="Thu, 15 Nov 2018 16:27:32 +0000"  >&lt;p&gt;Hi, I would like to know if you have an ETA for this feature.&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;</comment>
                            <comment id="2029190" author="eric.daniels@10gen.com" created="Wed, 10 Oct 2018 16:31:18 +0000"  >&lt;p&gt;I think from a sharded perspective we have to return the minimum of all the shards last cluster times. That will ensure that you can resume and not miss an event in the event that one shard is ahead. So for a replica set, it would use be the latest time.&lt;/p&gt;</comment>
                            <comment id="2029175" author="kevin.albertson" created="Wed, 10 Oct 2018 16:21:21 +0000"  >&lt;p&gt;Of the proposed solutions I vote for the first (exposing the latest optime the change stream cursor has observed) in the getMore and aggregate commands.&lt;/p&gt;

&lt;p&gt;If we instead relax the assertions, I assume the behavior would be to resume the change stream at the node&apos;s majority committed operation time. So we could still lose an event, right?&lt;/p&gt;</comment>
                            <comment id="2026391" author="david.dana" created="Mon, 8 Oct 2018 12:45:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rahul.singhania&quot; class=&quot;user-hover&quot; rel=&quot;rahul.singhania&quot;&gt;rahul.singhania&lt;/a&gt; FYI please track.&lt;/p&gt;</comment>
                            <comment id="2003339" author="charlie.swanson" created="Fri, 14 Sep 2018 16:57:15 +0000"  >&lt;p&gt;Flagging for scheduling - this has come up recently with stitch triggers and seems eligible as a quick win.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="644838">SERVER-38414</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="665095">DRIVERS-595</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="487337">SERVER-32895</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="676494">SERVER-39143</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10011"><![CDATA[Minor Change]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000cDdyrIAC, 500A000000cSB24IAG, 5002K00000dD5KXQA0]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 19 Sep 2018 22:38:20 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 12 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SPEC-1122'>SPEC-1122</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-38414'>SERVER-38414</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1276</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 12 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>david.dana@mongodb.com</customfieldvalue>
            <customfieldvalue>jodordonne</customfieldvalue>
            <customfieldvalue>eric.daniels@mongodb.com</customfieldvalue>
            <customfieldvalue>jonathan.green@mongodb.com</customfieldvalue>
            <customfieldvalue>kevin.albertson@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu17t3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8bnb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2672">Query 2019-01-14</customfieldvalue>
    <customfieldvalue id="2752">Query 2019-02-11</customfieldvalue>
    <customfieldvalue id="2788">Query 2019-02-25</customfieldvalue>
    <customfieldvalue id="2789">Query 2019-03-11</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11856"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0u2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>