<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:46:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-37724] Replication recovery: fassert when starting up mongod </title>
                <link>https://jira.mongodb.org/browse/SERVER-37724</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;It&apos;s a replica set, running on version 4.0.3 in k8s.&#160;&lt;/p&gt;

&lt;p&gt;I would like to both find out how to recover from this and actually understand how did it end up in this state in the first place.&#160;&lt;/p&gt;

&lt;p&gt;My understanding is that after any applied operation (on the primary) that change is reflected on the oplog as well. I believe these happen atomically, so client gets an OK write response if both of these succeed.&lt;/p&gt;

&lt;p&gt;However, this does not mean the data is persisted yet (nor the operations in the journal, if they are less than 128 KB &lt;a href=&quot;https://docs.mongodb.com/manual/core/journaling/#journaling-process&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;#anchor 1[]&lt;/a&gt;). It&apos;s persisted when the next checkpoint kicks in. So when a hard shutdown occur, and there is some data in the journal buffer, that&apos;s basically lost.&lt;/p&gt;

&lt;p&gt;Which is in correspondence with the docs: &apos;In between write operations, while the journal records remain in the WiredTiger buffers, updates can be lost following a hard shutdown of&#160;&lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongod/#bin.mongod&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;mongod&lt;/tt&gt;&lt;/a&gt;.&apos;&lt;/p&gt;

&lt;p&gt;Going back to the log `Applied op { : Timestamp(1539695503, 1) } not found. Top of oplog is { : Timestamp(1539695466, 1) }`. Now, I&apos;m not 100% where is the &apos;applied op&apos; coming from (is it from the last checkpoint? - probably yes), but it&apos;s an operation that is newer than the top of the oplog, which means that when it was persisted, the related document in the oplog wasn&apos;t.&#160;&lt;/p&gt;

&lt;p&gt;My rough guess is that the hard shutdown occurred exactly at the point of the checkpoint, which then has succeeded only partially.&lt;/p&gt;

&lt;p&gt;I&apos;ve been a user of mongo for years now, but never really had the need to dig deeper into its working. Any help is greatly appreciated!&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;...&lt;/p&gt;

&lt;p&gt;2018-10-24T09:08:43.547+0000 I FTDC &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Initializing full-time diagnostic data capture with directory &apos;/data/db/diagnostic.data&apos;&lt;br/&gt;
2018-10-24T09:08:43.550+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Rollback ID is 9&lt;br/&gt;
2018-10-24T09:08:43.550+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Recovering from stable timestamp: Timestamp(1539695503, 1) (top of oplog: { ts: Timestamp(1539695466, 1), t: 236 }, appliedThrough: { ts: Timestamp(1539695503, 1), t: 237 }, TruncateAfter: Timestamp(0, 0))&lt;br/&gt;
2018-10-24T09:08:43.550+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Starting recovery oplog application at the stable timestamp: Timestamp(1539695503, 1)&lt;br/&gt;
2018-10-24T09:08:43.550+0000 F REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Applied op { : Timestamp(1539695503, 1) } not found. Top of oplog is { : Timestamp(1539695466, 1) }.&lt;br/&gt;
2018-10-24T09:08:43.550+0000 F - &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Fatal Assertion 40313 at src/mongo/db/repl/replication_recovery.cpp 359&lt;br/&gt;
2018-10-24T09:08:43.550+0000 F - &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;***aborting after fassert() failure&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;#1&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;1&lt;/a&gt;-&#160;&lt;a href=&quot;https://docs.mongodb.com/manual/core/journaling/#journaling-process&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/core/journaling/#journaling-process&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="623415">SERVER-37724</key>
            <summary>Replication recovery: fassert when starting up mongod </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="Zoltan">Zoltan Mark Bodor [X]</reporter>
                        <labels>
                    </labels>
                <created>Wed, 24 Oct 2018 10:32:56 +0000</created>
                <updated>Thu, 3 Jan 2019 20:32:13 +0000</updated>
                            <resolved>Wed, 14 Nov 2018 15:25:42 +0000</resolved>
                                    <version>4.0.3</version>
                                                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="2104067" author="joachim" created="Thu, 3 Jan 2019 08:55:14 +0000"  >&lt;p&gt;Hi Daniel,&lt;/p&gt;

&lt;p&gt;Yes, my issue is also within a Kubernetes cluster.&lt;/p&gt;

&lt;p&gt;After I rolled back (5 days ago) to mongo version&#160;&#160;MongoDB 3.6 I have not seen the problem again.&#160;&lt;/p&gt;

&lt;p&gt;To your questions:&lt;/p&gt;

&lt;p&gt;&#160;&lt;b&gt;What is the configuration of the replica set?&lt;/b&gt; &#8211; This is the configuration we use now. before we used the latest mongo docker which caused the above-mentioned error&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/Chronasorg/chronas-api/blob/master/infrastructure/kubernetes/mongo/mongo-statefulset.yml&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/Chronasorg/chronas-api/blob/master/infrastructure/kubernetes/mongo/mongo-statefulset.yml&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Are your journal and data directories on different disks?&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;only&#160; /data/db&#160;is mounted to a different disk. I don&apos;t know where the journal is placed&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;When you say &quot;hard shutdown&quot;, what exactly happened in this situation?&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;The pod&#160;inside the kubernetes&#160;cluster was recreated the disc was staying unchanged. This happened before multiple times without&#160;any issue&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Please provide the&#160;&lt;tt&gt;mongod&lt;/tt&gt;&#160;logs for all the nodes in the replica set covering the timeframe before the crash so we can take a look.&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Unfortunately, as this was a critical moment and the system was not working anymore I have not stored the logs&#160;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2103698" author="daniel.hatcher" created="Wed, 2 Jan 2019 21:24:34 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=joachim&quot; class=&quot;user-hover&quot; rel=&quot;joachim&quot;&gt;joachim&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The error listed here and in that Stack Overflow post state that the newest entry in the oplog is older than the last operation applied by the storage engine which should not occur. Is your issue also within a Kubernetes cluster? If so, it may be related to how Kubernetes is configured.&lt;/p&gt;

&lt;p&gt;In order for us to troubleshoot on the MongoDB-side, please respond to the questions I posed in my &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37724?focusedCommentId=2043443&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2043443&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;previous comment&lt;/a&gt; and we can investigate further.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2101154" author="joachim" created="Thu, 27 Dec 2018 19:27:16 +0000"  >&lt;p&gt;I have the same issue described in this stack overvlow&#160;post &lt;a href=&quot;https://stackoverflow.com/questions/53835384/mongodb-statefulset-on-kubernetes-is-not-working-anymore-after-kubernetes-update&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://stackoverflow.com/questions/53835384/mongodb-statefulset-on-kubernetes-is-not-working-anymore-after-kubernetes-update&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2043443" author="daniel.hatcher" created="Fri, 26 Oct 2018 16:02:30 +0000"  >&lt;p&gt;Hello Zoltan,&lt;/p&gt;

&lt;p&gt;Your understanding of the way writes work is mostly correct. In 4.0, we did make a change to the journaling process in which it follows the oplog instead of the data files themselves. That has a chance to be what is happening here.&lt;/p&gt;

&lt;p&gt;This situation is certainly unexpected. Could you provide more context as to the configuration of your cluster and the situation preceding the shutdown?&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;What is the configuration of the replica set?&lt;/li&gt;
	&lt;li&gt;Are your journal and data directories on different disks?&lt;/li&gt;
	&lt;li&gt;When you say &quot;hard shutdown&quot;, what exactly happened in this situation?&lt;/li&gt;
	&lt;li&gt;Please provide the &lt;tt&gt;mongod&lt;/tt&gt; logs for all the nodes in the replica set covering the timeframe before the crash so we can take a look.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Danny&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="199320" name="mongo.log" size="10388" author="Zoltan" created="Wed, 24 Oct 2018 09:24:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 26 Oct 2018 16:02:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 5 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 5 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>joachim</customfieldvalue>
            <customfieldvalue>Zoltan</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hub3xr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu1dzj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huaq73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>