<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:14:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-27277] [rsBackgroundSync] Fatal assertion 18750 UnrecoverableRollbackError on numerous 3.2.10 replica sets</title>
                <link>https://jira.mongodb.org/browse/SERVER-27277</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We had a weekend full of networks issues which caused data replication all over our topology to become stale (slaves were lagging after masters etc.) &lt;/p&gt;

&lt;p&gt;All lags and discrepancies were shortly settled once the network issues were addressed (in databases such as MySQL, Cassandra)&lt;/p&gt;

&lt;p&gt;A couple of our mongo replica sets in production, however, failed to recover from the network outages, as can be seen in the log enclosed hereby, specifically for one of them, there appeared to be several attempts at re-election for primary followed by attempts to interconnect with other members which failed due to network unavailability and ignited a background sync rollback that finally has failed after several attempts &lt;br/&gt;
And eventually resulted the above mentioned Fatal assertion message.&lt;br/&gt;
Forcing us into restarting the primary replica and any other failing nodes in order to restore normal replication cross replica set.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-12-02T21:37:35.681+0000 I -        [rsBackgroundSync] Fatal assertion 18750 UnrecoverableRollbackError: need to rollback, but in inconsistent state. minvalid: (term: 21, timestamp: Dec  2 21:37:33:2) &amp;gt; our last optime: (term: 20, timestamp: Dec  2 21:37:32:1)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;As mentioned before, this has occurred on a couple of mission critical replica sets which failed to recover from it.&lt;br/&gt;
We noticed this behaviour might have to do with the mongo release version which all of our replica sets are on, which is 3.2.10.&lt;/p&gt;

&lt;p&gt;We&apos;ve upgraded from 3.2.9 to 3.2.10 after some harsh performance related bugs,&lt;br/&gt;
Shall we go ahead und upgrade our sets to 3.2.11 to resolve this?&lt;br/&gt;
Is thee another analysis you can provide and steps to be followed as recommendation?&lt;/p&gt;

&lt;p&gt;We really need those replica sets stable working on mongo, same as other resilient db solutions we have and that have recovered from this completely.&lt;/p&gt;

&lt;p&gt;Thanks for your prompt advise!&lt;br/&gt;
Enclosed as well is the diagnostics directory for that primary member in which the erroneous behaviour was spotted.&lt;/p&gt;

&lt;p&gt;Avi Kalvo&lt;br/&gt;
WiX DBA&lt;/p&gt;
</description>
                <environment></environment>
        <key id="336292">SERVER-27277</key>
            <summary>[rsBackgroundSync] Fatal assertion 18750 UnrecoverableRollbackError on numerous 3.2.10 replica sets</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="avrahamk">Avraham Kalvo</reporter>
                        <labels>
                    </labels>
                <created>Mon, 5 Dec 2016 13:19:18 +0000</created>
                <updated>Mon, 17 Jul 2017 15:40:30 +0000</updated>
                            <resolved>Tue, 6 Dec 2016 22:08:41 +0000</resolved>
                                    <version>3.2.10</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1450792" author="thomas.schubert" created="Tue, 6 Dec 2016 22:08:32 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=avrahamk&quot; class=&quot;user-hover&quot; rel=&quot;avrahamk&quot;&gt;avrahamk&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;While there have been a number a performance improvements in 3.2.11 that may impact the behavior you are observing, we cannot point to a particular ticket that would resolve the issue described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27132&quot; title=&quot;Mongo response time increased in 15-20% after upgrade to 3.2.10&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27132&quot;&gt;&lt;del&gt;SERVER-27132&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I&apos;ve examined the logs in more detail and are confident that you are hitting &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25145&quot; title=&quot;During rollback (or w/minvalid invalid) select sync source based on end OpTime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25145&quot;&gt;&lt;del&gt;SERVER-25145&lt;/del&gt;&lt;/a&gt;, and will be closing this ticket as a duplicate. If you encounter this issue again after upgrading, please open a new ticket and we will continue to investigate.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1450019" author="avrahamk" created="Tue, 6 Dec 2016 09:18:36 +0000"  >&lt;p&gt;Thanks Ramon,&lt;/p&gt;

&lt;p&gt;Can you also confirm 3.2.11 will resolve performance issues as reported by our end on another ticket with you &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27132&quot; title=&quot;Mongo response time increased in 15-20% after upgrade to 3.2.10&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27132&quot;&gt;&lt;del&gt;SERVER-27132&lt;/del&gt;&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Avi K &lt;/p&gt;</comment>
                            <comment id="1449526" author="ramon.fernandez" created="Mon, 5 Dec 2016 18:51:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=avrahamk&quot; class=&quot;user-hover&quot; rel=&quot;avrahamk&quot;&gt;avrahamk&lt;/a&gt;, the log line you put in the description seems to indicate that you&apos;re running into &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25145&quot; title=&quot;During rollback (or w/minvalid invalid) select sync source based on end OpTime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25145&quot;&gt;&lt;del&gt;SERVER-25145&lt;/del&gt;&lt;/a&gt;, which was fixed in 3.2.11, so I&apos;d recommend upgrading to 3.2.11 and report back.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="302251">SERVER-25145</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="145317" name="mongodb_shoutout_subs.tar.gz" size="10452809" author="avrahamk" created="Mon, 5 Dec 2016 13:14:09 +0000"/>
                            <attachment id="145316" name="mongodb_shoutout_subs_diagnostic_data.tar.gz" size="17270581" author="avrahamk" created="Mon, 5 Dec 2016 13:17:41 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000WUZehIAH]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 5 Dec 2016 18:51:23 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 10 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>benety.goh@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 10 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>avrahamk</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjpev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hssitj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4n5z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>