<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:14:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7286] Can I access data that seems to be stored in my local.* files</title>
                <link>https://jira.mongodb.org/browse/SERVER-7286</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I badly need to access the replication op log after a serious data corruption that occurred while trying to copy databases on the same server.&lt;/p&gt;

&lt;p&gt;I have the following 5 local files in the current db path.&lt;/p&gt;

&lt;p&gt;64M		local.0&lt;br/&gt;
128M		local.1&lt;br/&gt;
2.0G		local.2&lt;br/&gt;
2.0G		local.3&lt;br/&gt;
2.0G		local.4&lt;/p&gt;

&lt;p&gt;When I acces the &quot;local&quot; db via the mongo shell and the according &quot;oplog.rs&quot; collection, I only see 3 entries. Prior to the crash, I had upwards of 600,000 entries and so am hoping they have been orphaned in the local.1 - local.4 files. &lt;/p&gt;

&lt;p&gt;Is there any that I can verify whether these rows still exist and, if so, how might I access them?&lt;/p&gt;

&lt;p&gt;Thanks for any help you can provide.&lt;/p&gt;

&lt;p&gt;-Robert Poskevich&lt;/p&gt;</description>
                <environment>Linux Debian Squeeze</environment>
        <key id="52520">SERVER-7286</key>
            <summary>Can I access data that seems to be stored in my local.* files</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="gregor">Gregor Macadam</assignee>
                                    <reporter username="rposky">Robert Poskevich</reporter>
                        <labels>
                            <label>crash</label>
                            <label>oplog</label>
                            <label>replication</label>
                    </labels>
                <created>Sat, 6 Oct 2012 20:35:15 +0000</created>
                <updated>Fri, 8 Mar 2013 15:55:58 +0000</updated>
                            <resolved>Mon, 12 Nov 2012 16:22:13 +0000</resolved>
                                    <version>2.0.4</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="186647" author="gregor" created="Mon, 12 Nov 2012 16:22:00 +0000"  >&lt;p&gt;Closing this issue out. Please feel free to reopen if you have more questions or when you have time to upload the logs. &lt;br/&gt;
Thanks&lt;br/&gt;
Gregor&lt;/p&gt;</comment>
                            <comment id="180868" author="gregor" created="Wed, 31 Oct 2012 10:18:57 +0000"  >&lt;p&gt;Apologies for the delay in coming back to you - in log we can see that mongos1r1.qmobix.net:27018 becomes PRIMARY - this is the the SECONDARY member that you are referring to? I think the reason it has been elected PRIMARY is that the host domU-12-31-39-07-B6-7C.compute-1.internal has an empty oplog. I haven&apos;t been able to repro this unfortunately - could you attach more of the log from prior to the initial sync - perhaps it shows what happened to the oplog. &lt;br/&gt;
Thanks&lt;br/&gt;
Gregor&lt;/p&gt;</comment>
                            <comment id="178190" author="rposky" created="Tue, 23 Oct 2012 15:41:40 +0000"  >&lt;p&gt;Gregor,&lt;/p&gt;

&lt;p&gt;In performing the copy, Y was on the same host.&lt;/p&gt;

&lt;p&gt;Do you have any idea what could be the cause of the my last stated issue? &quot;The SECONDARY member which became the MASTER in this scenario was 3 months behind.&quot;&lt;/p&gt;</comment>
                            <comment id="176690" author="gregor" created="Thu, 18 Oct 2012 16:29:52 +0000"  >&lt;p&gt;A resync takes place if the local oplog is empty - the code that does this is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/rs_sync.cpp#L664&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;br/&gt;
That doesn&apos;t explain how the oplog came to be empty. I want to try to repro this here - when you did the copy database you were, Y was on a the same host or a different host?&lt;/p&gt;</comment>
                            <comment id="173762" author="rposky" created="Thu, 11 Oct 2012 15:12:05 +0000"  >&lt;p&gt;Gregor,&lt;/p&gt;

&lt;p&gt;The above portion of the log occurred after the copy command. The copy command was not documented in the log but you can see, in the very first line of the attached log, when I attempted to kill the mongod process.&lt;/p&gt;

&lt;p&gt;What are the scenarios in which a replica set member will undergo an initial sync? Can this only occur if the dbpath directory is empty? Are there other scenarios in which this might occur?&lt;/p&gt;

&lt;p&gt;Also, at the time, the SECONDARY member which became the MASTER in this scenario was 3 months behind. Can you think of why this might have been? That&apos;s the problem I really need to solve, so that I can avoid it happening and losing my data again.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Robert P.&lt;/p&gt;</comment>
                            <comment id="173655" author="gregor" created="Thu, 11 Oct 2012 07:50:15 +0000"  >&lt;p&gt;Hi - the log shows that an initial sync is taking place for this node&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] replSet initial sync pending&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] replSet syncing to: mongos1r1.qmobix.net:27018&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] build index local.me { _id: 1 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] build index done 0 records 0 secs&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] replSet initial sync drop all databases&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] dropAllDatabasesExceptLocal 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Fri Oct  5 23:23:15 [rsSync] replSet initial sync clone all databases&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;How is this related to the problems you had with copyDatabase? I&apos;m not clear what you are doing. &lt;/p&gt;</comment>
                            <comment id="173367" author="rposky" created="Wed, 10 Oct 2012 16:01:10 +0000"  >&lt;p&gt;In, examining the log, it looks as if all databased were dropped in the synching process. However, in performing the mongodump with journaling enabled (as journal files are present), I only get 6 entries, all dated after the incident.&lt;/p&gt;</comment>
                            <comment id="173353" author="rposky" created="Wed, 10 Oct 2012 15:36:40 +0000"  >&lt;p&gt;MongoDB Log - Log activity prior to the first line pasted seems normal and consists of authentication checks. Log activity after the last line pasted is a sync with the MASTER node of all dbs, which looks to be successful. After the sync, the node assumes SECONDARY status and proceeds to make authentication requests. Unfortunately at the time, the MASTER was 3 months out of date. How can this be?&lt;/p&gt;</comment>
                            <comment id="173261" author="gregor" created="Wed, 10 Oct 2012 10:33:17 +0000"  >&lt;p&gt;Do you have the mongod log from where you ran the copyDatabase?&lt;br/&gt;
As far as recovering the oplog entries - can you try to start a standalone mongodb instance and point it at the data files.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./mongod --dbpath &amp;lt;path_to_datafiles&amp;gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;and then run a &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongodump --db local --repair&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;to dump out the local database and paste the output please.&lt;/p&gt;</comment>
                            <comment id="172303" author="rposky" created="Sat, 6 Oct 2012 21:12:38 +0000"  >&lt;p&gt;Eliot,&lt;br/&gt;
Thank you, I have copied the files. The data corruption occurred when attempting to copy a database, let&apos;s say &quot;X&quot; is the original and &quot;Y&quot; is the clone.&lt;/p&gt;

&lt;p&gt;db.fsyncLock()&lt;br/&gt;
db.copyDatabase(X, Y)&lt;/p&gt;

&lt;p&gt;I let the operation run for ~20 minutes (much longer than other copy operations on the same dataset) and, at that point, closed my ssh connection. Upon reconnecting and accessing the shell, I found that both X and Y were empty, save for the &quot;system.indexes&quot; and &quot;system.users&quot; collections. I then attempted to shutdown the server and launch from a backup (I am running on Amazon EC2), only to find that the data in my backup was incomplete and was missing ~3 months of activity. And storage is not the issue here, as I&apos;ve only consumed some 14% of the available.&lt;/p&gt;

&lt;p&gt;I am running a single sharded system, with 2 replica nodes and an arbiter as well as 3 config instances. The node that crashed was the current PRIMARY, while the snapshots were taken off of the SECONDARY. The SECONDARY node, at this time and throughout the snapshot history, is missing X entirely, as well as any oplog history within the last 3 months. I hadn&apos;t checked the replication status in some time, though, the last time I did, all three nodes were healthy. &lt;/p&gt;

&lt;p&gt;My intention in using fsync lock was to disallow any writes to X while I was copying to Y, but I think this could have been the cause for the initial process hang. I have disconnected the server in question from the replica set (the one with the listed local.* files) and have launched it as a single instance. Perhaps that is the reason I have two local.* files less than 2 GB.&lt;/p&gt;

&lt;p&gt;I hope that description can help. Thanks for your attention.&lt;/p&gt;</comment>
                            <comment id="172301" author="eliot" created="Sat, 6 Oct 2012 20:47:14 +0000"  >&lt;p&gt;Can you describe what happened to cause the issue and what steps you took?&lt;br/&gt;
I would also recommend making a copy of the files immediately.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="20589" name="crash_log.txt" size="11625" author="rposky" created="Wed, 10 Oct 2012 15:36:40 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sat, 6 Oct 2012 20:47:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 14 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 14 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>gregor</customfieldvalue>
            <customfieldvalue>rposky</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnlz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrjv3j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29572</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht05l3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>