<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:51:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-61168] Mongo Secondary Oplog Keeps Getting Too Far Behind Primary</title>
                <link>https://jira.mongodb.org/browse/SERVER-61168</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a replica set comprising of a primary, secondary and arbiter. Our database sizes are 20TB+ and we have set our Oplog sizing so that it was 16 hours long the first time our secondary got too far behind. After repriming the secondary and extending the Oplog to 54 hours, we went another 3 weeks before we experienced this issue again over the weekend.&lt;/p&gt;

&lt;p&gt;What specific things should we look for at the culprit for this? From documentation, it looks like disk I/O or network issues could potentially cause these issues but I&apos;m not seeing any indication of that so far but really just want to check all of our bases before looking into that any more.&lt;/p&gt;

&lt;p&gt;Would sharding out the database help with issues like this? Our database growth keeps climbing and we definitely need to start looking to see if sharding can solve this issue.&lt;/p&gt;

&lt;p&gt;I can upload logs too if necessary. Thanks&lt;/p&gt;</description>
                <environment></environment>
        <key id="1914753">SERVER-61168</key>
            <summary>Mongo Secondary Oplog Keeps Getting Too Far Behind Primary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="edwin.zhou@mongodb.com">Edwin Zhou</assignee>
                                    <reporter username="ncallen@cobaltiron.com">Neil Allen</reporter>
                        <labels>
                    </labels>
                <created>Mon, 1 Nov 2021 17:34:56 +0000</created>
                <updated>Thu, 4 Nov 2021 21:02:45 +0000</updated>
                            <resolved>Thu, 4 Nov 2021 21:02:45 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4170421" author="JIRAUSER1257066" created="Thu, 4 Nov 2021 21:02:45 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ncallen%40cobaltiron.com&quot; class=&quot;user-hover&quot; rel=&quot;ncallen@cobaltiron.com&quot;&gt;ncallen@cobaltiron.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;After inspecting the log files and FTDC, it appears that the secondary node goes into ROLLBACK but is unable to catch up because the primary oplog is overflowing. &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-30T06:46:33.183Z&quot;},&quot;s&quot;:&quot;I&quot;,&quot;c&quot;:&quot;ROLLBACK&quot;,&quot;id&quot;:21681,&quot;ctx&quot;:&quot;BackgroundSync&quot;,&quot;msg&quot;:&quot;Starting rollback&quot;,&quot;attr&quot;:{&quot;syncSource&quot;:&quot;cspdb04ip.us2.cobaltiron.com:27017&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-30T06:46:33.422Z&quot;},&quot;s&quot;:&quot;I&quot;,&quot;c&quot;:&quot;ROLLBACK&quot;,&quot;id&quot;:21682,&quot;ctx&quot;:&quot;BackgroundSync&quot;,&quot;msg&quot;:&quot;Finding the Common Point&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-30T11:15:06.992Z&quot;},&quot;s&quot;:&quot;W&quot;,&quot;c&quot;:&quot;ROLLBACK&quot;,&quot;id&quot;:21728,&quot;ctx&quot;:&quot;BackgroundSync&quot;,&quot;msg&quot;:&quot;Rollback cannot complete at this time (retrying later)&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;CappedPositionLost: CollectionScan died due to position in capped collection being deleted. Last seen record id: RecordId(7024107236637019329)&quot;,&quot;myLastAppliedOpTime&quot;:{&quot;ts&quot;:{&quot;_timestamp&quot;:{&quot;t&quot;:1635411709,&quot;i&quot;:1973}},&quot;t&quot;:18},&quot;minValid&quot;:{&quot;ts&quot;:{&quot;_timestamp&quot;:{&quot;t&quot;:1635411709,&quot;i&quot;:1973}},&quot;t&quot;:18}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-30T11:15:06.992Z&quot;},&quot;s&quot;:&quot;F&quot;,&quot;c&quot;:&quot;ROLLBACK&quot;,&quot;id&quot;:4655800,&quot;ctx&quot;:&quot;BackgroundSync&quot;,&quot;msg&quot;:&quot;Index builds stopped prior to rollback cannot be restarted by subsequent rollback attempts&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;You may be able to find guidance in our docs to help&lt;a href=&quot;https://docs.mongodb.com/manual/core/replica-set-rollbacks/#avoid-replica-set-rollbacks&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;avoid replica set rollbacks&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The SERVER project is reserved for bug reports, so if you&apos;d like to troubleshoot this further we&apos;d like to encourage you to start by asking our community for help by posting on the &lt;a href=&quot;https://community.mongodb.com&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB Developer Community Forums&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If the discussion there leads you to suspect a bug in the MongoDB server, then we&apos;d want to investigate it as a possible bug here in the SERVER project.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4165241" author="JIRAUSER1263146" created="Tue, 2 Nov 2021 21:18:45 +0000"  >&lt;p&gt;Logs have been uploaded to support uploader as&#160;cspdb03ip-2021-10-30.log.bz2 (mongo log from day of crash) and diagnostic data&#160;cspdb03ip-diagnostic-data.tar.gz&lt;/p&gt;</comment>
                            <comment id="4164679" author="JIRAUSER1257066" created="Tue, 2 Nov 2021 18:35:56 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ncallen%40cobaltiron.com&quot; class=&quot;user-hover&quot; rel=&quot;ncallen@cobaltiron.com&quot;&gt;ncallen@cobaltiron.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for you for your report. Would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNDkzNDAzNDEwNDYiLCJleHAiOjE2Mzg0NzAxMjV9.wbqQYGAi-w6AHQm_TQ8qYnXvXjs-XU2UA4DnL9J_Tls&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 2 Nov 2021 18:35:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 13 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 13 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>ncallen@cobaltiron.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i08j3j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzs0zz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0858v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>