<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:32:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-13654] InitialSync Replication over IPSEC link results in data corruption</title>
                <link>https://jira.mongodb.org/browse/SERVER-13654</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When attempting to replicate a secondary member over the internet through an IPSEC link, the following messages are found in the logs in the secondary server during initial sync:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; Cloner: skipping corrupt object from testdb.testColl1 firstElement: _id: ObjectId(&apos;533440f68fde6a4ce2c4941e&apos;)&lt;/p&gt;

&lt;p&gt;These messages only occur when replicating over the internet, through the IPSEC link.  When replicating locally, there is no issue.&lt;/p&gt;

&lt;p&gt;The initial sync succeeds, however the new secondary is missing records from the primary.&lt;/p&gt;

&lt;p&gt;Primary&apos;s testdb.testColl1.count():&lt;br/&gt;
113501&lt;/p&gt;

&lt;p&gt;Secondary&apos;s testdb.testColl1.count():&lt;br/&gt;
113490&lt;/p&gt;

&lt;p&gt;I have tested the reportedly corrupted objects by retrieving them from the primary, and can retrieve them fine.&lt;/p&gt;</description>
                <environment>Ubuntu 13.10&lt;br/&gt;
Linux 3.11.0-12-generic&lt;br/&gt;
MongoDB 2.4.10</environment>
        <key id="131978">SERVER-13654</key>
            <summary>InitialSync Replication over IPSEC link results in data corruption</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="bbernstein@railkey.net">Brian Bernstein</reporter>
                        <labels>
                    </labels>
                <created>Fri, 18 Apr 2014 14:24:25 +0000</created>
                <updated>Wed, 10 Dec 2014 23:10:54 +0000</updated>
                            <resolved>Tue, 10 Jun 2014 21:20:09 +0000</resolved>
                                    <version>2.4.10</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="616149" author="ramon.fernandez" created="Tue, 10 Jun 2014 21:18:51 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bbernstein%40railkey.net&quot; class=&quot;user-hover&quot; rel=&quot;bbernstein@railkey.net&quot;&gt;bbernstein@railkey.net&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;we haven&apos;t heard back from you for some time. Is there any further information you want to add to this ticket? To add to David&apos;s answer, if the IPSEC tunnel is truncating or dropping packets, then the BSON data is bound to get corrupted &amp;#8211; at that point &lt;tt&gt;mongod&lt;/tt&gt; refuses to insert it in the database.&lt;/p&gt;

&lt;p&gt;In the past I&apos;ve run into issues with applications running over IPSEC tunnels where the solution was to lower the MTU, but this wasn&apos;t with MongoDB so your mileage may vary.&lt;/p&gt;

&lt;p&gt;At this stage it doesn&apos;t look like there&apos;s a bug in MongoDB, so I&apos;m going to mark this issue as resolved. If you believe otherwise feel free to re-open the ticket and provide additional information.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="578870" author="david.hows" created="Fri, 9 May 2014 23:17:24 +0000"  >&lt;p&gt;Hi Brian,&lt;/p&gt;

&lt;p&gt;I think you may be on the money. If the IPSEC tunnels are truncating packets or are truncating packets unexpectedly or silently dropping packets (silent as in causing them to appear as having never been sent) this could potentially cause problems.&lt;/p&gt;

&lt;p&gt;If the IPSEC tunnels are truncating the messages such that they are not &quot;obviously&quot; truncated (like not having missing sequence numbers for example) this could theoretically cause the issue.&lt;/p&gt;

&lt;p&gt;This was what i wished to investigate with the packet captures.&lt;/p&gt;

&lt;p&gt;Let me know how you go.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="578406" author="bbernstein@railkey.net" created="Fri, 9 May 2014 17:10:22 +0000"  >&lt;p&gt;I am currently trying to figure out how to capture the network packets before the IPSEC encapsulation.  The IPSEC tunnelling is occurring on the respective MongoDB&apos;s local hosts at the kernel level.  Unfortunately, I think that simply running tcpdump on these instances will result in captures after the IPSEC encryption.&lt;/p&gt;

&lt;p&gt;In the mean time, I&apos;ve examined the logs from the IPSEC replication and compared them to logs generated when cloning the initial data set to the test environment.&lt;/p&gt;

&lt;p&gt;In addition to the corrupt object messages, I&apos;ve noticed a lot of these messages in the IPSEC replication logs that are not present in the logs when cloning over the local network:&lt;/p&gt;

&lt;p&gt;Socket recv() conn closed? 54.86.48.186:43237&lt;br/&gt;
SocketException: remote: 54.86.48.186:43237 error: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;CLOSED&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;54.86.48.186:43237&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;These messages appear to suggest that the connection between the two instances are being interrupted at some point.  Is it possible that these connection errors are causing the corrupt object issues?&lt;/p&gt;

&lt;p&gt;Could the corruption be due to messages between the two servers getting truncated unexpectedly?&lt;/p&gt;</comment>
                            <comment id="578232" author="david.hows" created="Fri, 9 May 2014 14:58:17 +0000"  >&lt;p&gt;Hi Brian,&lt;/p&gt;

&lt;p&gt;MongoDB shouldn&apos;t have any direct interactions with the IPSEC itself (that should be handled lower in the stack) so I cant see how this would effect things directly.&lt;/p&gt;

&lt;p&gt;With this in mind would you be able to gather a few datapoints for us to investigate this better. Next time you run the resync over your IPSEC tunnel can you please perform some packet captures at either end outside the tunnel. I&apos;d like to see&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;What the from side sends&lt;/li&gt;
	&lt;li&gt;What the to side receives&lt;/li&gt;
	&lt;li&gt;What the syncing member reports&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Hopefully between those three pieces of data we can work out exactly what is going on and from there what is going wrong.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="557559" author="bbernstein@railkey.net" created="Mon, 21 Apr 2014 16:21:15 +0000"  >&lt;p&gt;Log file of secondary instance during initial sync.&lt;/p&gt;</comment>
                            <comment id="557557" author="bbernstein@railkey.net" created="Mon, 21 Apr 2014 16:20:33 +0000"  >&lt;p&gt;I&apos;ve repeated my replication testing across the link and the number of corrupted documents varies during each attempt.&lt;/p&gt;

&lt;p&gt;Out of a collection size of 113501, the number of corrupted documents after replication will vary between 8 to 13 documents.&lt;/p&gt;

&lt;p&gt;I had set the log level to 4 in one of the replication attempts.  I&apos;ll attach the log file to this issue.&lt;/p&gt;</comment>
                            <comment id="555840" author="bbernstein@railkey.net" created="Sat, 19 Apr 2014 16:12:45 +0000"  >&lt;p&gt;On the second attempt to perform the initial sync over the IPSec link, the logs contained different ObjectId&apos;s compared to the first attempt.  However, the number of corrupted documents appeared to be the same-- the respective counts for the primary and secondary instances of that collection did not change.&lt;/p&gt;

&lt;p&gt;I performed a simple test of the IPSec link to ensure it was working properly.  I transferred a 100MB file across the IPSec link with netcat and compared the md5sum at both ends.  The test worked successfully.&lt;/p&gt;</comment>
                            <comment id="555718" author="eliot" created="Sat, 19 Apr 2014 12:34:48 +0000"  >&lt;p&gt;Is it always the same documents or are they different?&lt;br/&gt;
Have you tested the ipsec link in general to make sure its clean?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="41855" name="mongodb.log" size="187437" author="bbernstein@railkey.net" created="Mon, 21 Apr 2014 16:21:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sat, 19 Apr 2014 12:34:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 36 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 36 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bbernstein@railkey.net</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlwin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hry527:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>113330</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsgyuv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>