<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:02:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3096] replicaet data integrity fail</title>
                <link>https://jira.mongodb.org/browse/SERVER-3096</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Critical bug in replica set synchronisation.&lt;/p&gt;

&lt;p&gt;I&apos;ve runned repairDatabase() to compact database files on my replSet primary host(HOST1).&lt;br/&gt;
After running that command server has exhausted resources (full mem and swap use).&lt;br/&gt;
MongoDB was restarted. Other host (HOST2) in replSet was set to primary status.&lt;/p&gt;

&lt;p&gt;When I check &lt;a href=&quot;http://ANYHOST:28017/_replSet&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://ANYHOST:28017/_replSet&lt;/a&gt; I found that optime on all nodes is the same.&lt;br/&gt;
I&apos;ve restarted primary host and HOST1 became primary.&lt;/p&gt;

&lt;p&gt;After logging to mongo cli and running command &quot;show dbs&quot;; I&apos;ve seen that two first dbs was showned as &quot;empty&quot;.&lt;/p&gt;</description>
                <environment>linux (openvz kernel without limit on vhosts, vserver kernel, native linux kernel), mongodb1.8</environment>
        <key id="17118">SERVER-3096</key>
            <summary>replicaet data integrity fail</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="mathias@mongodb.com">Mathias Stearn</assignee>
                                    <reporter username="msz">MartinS</reporter>
                        <labels>
                    </labels>
                <created>Mon, 16 May 2011 08:49:46 +0000</created>
                <updated>Mon, 29 Aug 2011 15:56:47 +0000</updated>
                            <resolved>Thu, 4 Aug 2011 16:34:38 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="46391" author="redbeard0531" created="Thu, 4 Aug 2011 16:34:38 +0000"  >&lt;p&gt;Please reopen if you have more information to help us track this down&lt;/p&gt;</comment>
                            <comment id="38497" author="redbeard0531" created="Tue, 21 Jun 2011 15:15:55 +0000"  >&lt;p&gt;Do you still have the log file from when mongod crashed with --journal? Have you seen this with 1.8.2?&lt;/p&gt;</comment>
                            <comment id="33255" author="msz" created="Wed, 18 May 2011 10:32:56 +0000"  >&lt;p&gt;Oh, and one more - after crash failcounter in openvz UBC was not incremented. Bug in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1121&quot; title=&quot;Crashes reported on OpenVZ&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1121&quot;&gt;&lt;del&gt;SERVER-1121&lt;/del&gt;&lt;/a&gt; affects only vhost vith memory limits and failcnt will be increased when virtual memory limit is exhausted.&lt;/p&gt;</comment>
                            <comment id="32942" author="msz" created="Mon, 16 May 2011 18:06:11 +0000"  >&lt;p&gt;&amp;gt; Which version of mongo exactly?&lt;/p&gt;

&lt;p&gt;1.8.1&lt;/p&gt;

&lt;p&gt;&amp;gt; Also, there are some known issues on openvz with data sets larger than ram due to an openvz issue.&lt;br/&gt;
&amp;gt; See &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1121&quot; title=&quot;Crashes reported on OpenVZ&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1121&quot;&gt;&lt;del&gt;SERVER-1121&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I know something about this. I&apos;ve tested it. This bug affect only virtualhost with memory limit set, I don&apos;t use theese limit. Server has exausted comletely host RAM and swap.&lt;/p&gt;

&lt;p&gt;I also know this bug with memory limit&apos;s doesn&apos;t affect vserver virtualisation (where memory limits are above 4GB).&lt;/p&gt;</comment>
                            <comment id="32941" author="eliot" created="Mon, 16 May 2011 18:01:15 +0000"  >&lt;p&gt;Which version of mongo exactly?  &lt;br/&gt;
Also, there are some known issues on openvz with data sets larger than ram due to an openvz issue.&lt;br/&gt;
See &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1121&quot; title=&quot;Crashes reported on OpenVZ&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1121&quot;&gt;&lt;del&gt;SERVER-1121&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="32940" author="msz" created="Mon, 16 May 2011 17:57:38 +0000"  >&lt;p&gt;Maybe. Unfortunately I can&apos;t force this behaviour again &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Oh, and this could be important: first two databases: admin and x were empty (both has 200MB in &apos;show dbs;&apos;), but i&apos;ve ran repairDatabase(); on the third database, bigger than those two.&lt;/p&gt;

&lt;p&gt;I regret that I haven&apos;t made snapshot of the filesystem after crash :/&lt;/p&gt;</comment>
                            <comment id="32936" author="kristina" created="Mon, 16 May 2011 17:48:06 +0000"  >&lt;p&gt;Sounds like a journaling issue?&lt;/p&gt;</comment>
                            <comment id="32935" author="kristina" created="Mon, 16 May 2011 17:47:42 +0000"  >&lt;p&gt;&amp;gt; I&apos;m using --journal already and this crash take place exactly in this configurations. After restart two databases are emtpy. &lt;/p&gt;

&lt;p&gt;Oh, okay, that&apos;s not good.  &lt;/p&gt;</comment>
                            <comment id="32933" author="msz" created="Mon, 16 May 2011 17:44:30 +0000"  >&lt;p&gt;&amp;gt; &lt;br/&gt;
&amp;gt; In logfile i didnt found any messages about running out of resources.&lt;br/&gt;
&amp;gt;&lt;br/&gt;
&amp;gt; Can you paste the end of your log here? (Last couple hundred lines or so, the more the better.)&lt;br/&gt;
&amp;gt;&lt;/p&gt;

&lt;p&gt;in the logfile there is only:&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;dur&amp;#93;&lt;/span&gt; lsn set &lt;span class=&quot;error&quot;&gt;&amp;#91;connection no&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from &lt;span class=&quot;error&quot;&gt;&amp;#91;IP&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;conn ID&amp;#93;&lt;/span&gt; end connection &lt;span class=&quot;error&quot;&gt;&amp;#91;IP&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;nothing else.&lt;/p&gt;

&lt;p&gt;After restart in logfile are lines like:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;conn$ID&amp;#93;&lt;/span&gt; assertion 10057 unauthorized db:dbname lock type:-1 client:IP ns:database.system.namespaces query:{}&lt;/p&gt;

&lt;p&gt;&amp;gt;&amp;gt; Server should never became primary host in this situation.&lt;br/&gt;
&amp;gt;&lt;br/&gt;
&amp;gt; MongoDB cannot tell if its data is corrupt or not without checking each record, which is why you can&apos;t restart it without deleting the mongod.lock file first. &lt;br/&gt;
&amp;gt; If you delete the lock file and restart, you&apos;re essentially telling MongoDB, &quot;Even though you think this data might be corrupt, I know it isn&apos;t. Use it anyway.&quot;&lt;br/&gt;
&amp;gt; If you don&apos;t want this situation, don&apos;t delete the mongod.lock file or use --journal.&lt;/p&gt;

&lt;p&gt;I&apos;m using --journal already and this crash take place exactly in this configurations. After restart two databases are emtpy.&lt;/p&gt;</comment>
                            <comment id="32909" author="kristina" created="Mon, 16 May 2011 15:41:30 +0000"  >&lt;p&gt;&amp;gt; I&apos;d rather to report a bug. MongoDB server should throw exception and handle it istead of crashing. &lt;/p&gt;

&lt;p&gt;Agreed.&lt;/p&gt;

&lt;p&gt;&amp;gt; In logfile i didnt found any messages about running out of resources. &lt;/p&gt;

&lt;p&gt;Can you paste the end of your log here?  (Last couple hundred lines or so, the more the better.)&lt;/p&gt;

&lt;p&gt;&amp;gt; Server should never became primary host in this situation. &lt;/p&gt;

&lt;p&gt;MongoDB cannot tell if its data is corrupt or not without checking each record, which is why you can&apos;t restart it without deleting the mongod.lock file first.  If you delete the lock file and restart, you&apos;re essentially telling MongoDB, &quot;Even though you think this data might be corrupt, I know it isn&apos;t.  Use it anyway.&quot;  If you don&apos;t want this situation, don&apos;t delete the mongod.lock file or use --journal.&lt;/p&gt;</comment>
                            <comment id="32907" author="msz" created="Mon, 16 May 2011 15:33:21 +0000"  >&lt;p&gt;&amp;gt; This is generally a good sign, they should be the same or very close...&lt;br/&gt;
&amp;gt;&lt;br/&gt;
&amp;gt; It sounds like HOST1 is corrupt, probably from running out of resources on the repair (did it crash?). On the plus side, it&apos;s very unlikely that replication will &lt;br/&gt;
&amp;gt; replicate corruption, so the best thing would be to:&lt;br/&gt;
&amp;gt;&lt;br/&gt;
&amp;gt; 1) connect to HOST2&lt;br/&gt;
&amp;gt; 2) make sure the dbs you expect are there&lt;br/&gt;
&amp;gt; 3) shut down HOST1, delete the data files, resync it from HOST2 or fastsync from a backup &lt;/p&gt;

&lt;p&gt;I&apos;m not looking for temporary solution for this problem. I done this 3 steps immediately after crash.&lt;/p&gt;

&lt;p&gt;I&apos;d rather to report a bug. MongoDB server should throw exception and handle it istead of crashing.&lt;br/&gt;
In logfile i didnt found any messages about running out of resources. &lt;/p&gt;

&lt;p&gt;Server should never became primary host in this situation.&lt;br/&gt;
MongoDB with this bug could be used only for cache data, not as main database.&lt;/p&gt;</comment>
                            <comment id="32905" author="kristina" created="Mon, 16 May 2011 15:18:13 +0000"  >&lt;p&gt;&amp;gt; When I check &lt;a href=&quot;http://ANYHOST:28017/_replSet&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://ANYHOST:28017/_replSet&lt;/a&gt; I found that optime on all nodes is the same.&lt;/p&gt;

&lt;p&gt;This is generally a good sign, they should be the same or very close...&lt;/p&gt;

&lt;p&gt;It sounds like HOST1 is corrupt, probably from running out of resources on the repair (did it crash?).  On the plus side, it&apos;s very unlikely that replication will replicate corruption, so the best thing would be to:&lt;/p&gt;

&lt;p&gt;1) connect to HOST2&lt;br/&gt;
2) make sure the dbs you expect are there&lt;br/&gt;
3) shut down HOST1, delete the data files, resync it from HOST2 or fastsync from a backup&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 16 May 2011 15:18:13 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 28 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 28 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>msz</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrozt3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriffr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21153</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0er3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>