[SERVER-13654] InitialSync Replication over IPSEC link results in data corruption Created: 18/Apr/14  Updated: 10/Dec/14  Resolved: 10/Jun/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Brian Bernstein Assignee: David Hows
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 13.10
Linux 3.11.0-12-generic
MongoDB 2.4.10


Attachments: Text File mongodb.log    
Operating System: Linux
Participants:

 Description   

When attempting to replicate a secondary member over the internet through an IPSEC link, the following messages are found in the logs in the secondary server during initial sync:

[rsSync] Cloner: skipping corrupt object from testdb.testColl1 firstElement: _id: ObjectId('533440f68fde6a4ce2c4941e')

These messages only occur when replicating over the internet, through the IPSEC link. When replicating locally, there is no issue.

The initial sync succeeds, however the new secondary is missing records from the primary.

Primary's testdb.testColl1.count():
113501

Secondary's testdb.testColl1.count():
113490

I have tested the reportedly corrupted objects by retrieving them from the primary, and can retrieve them fine.



 Comments   
Comment by Ramon Fernandez Marina [ 10/Jun/14 ]

Hi bbernstein@railkey.net,

we haven't heard back from you for some time. Is there any further information you want to add to this ticket? To add to David's answer, if the IPSEC tunnel is truncating or dropping packets, then the BSON data is bound to get corrupted – at that point mongod refuses to insert it in the database.

In the past I've run into issues with applications running over IPSEC tunnels where the solution was to lower the MTU, but this wasn't with MongoDB so your mileage may vary.

At this stage it doesn't look like there's a bug in MongoDB, so I'm going to mark this issue as resolved. If you believe otherwise feel free to re-open the ticket and provide additional information.

Regards,
Ramón.

Comment by David Hows [ 09/May/14 ]

Hi Brian,

I think you may be on the money. If the IPSEC tunnels are truncating packets or are truncating packets unexpectedly or silently dropping packets (silent as in causing them to appear as having never been sent) this could potentially cause problems.

If the IPSEC tunnels are truncating the messages such that they are not "obviously" truncated (like not having missing sequence numbers for example) this could theoretically cause the issue.

This was what i wished to investigate with the packet captures.

Let me know how you go.

Thanks,
David

Comment by Brian Bernstein [ 09/May/14 ]

I am currently trying to figure out how to capture the network packets before the IPSEC encapsulation. The IPSEC tunnelling is occurring on the respective MongoDB's local hosts at the kernel level. Unfortunately, I think that simply running tcpdump on these instances will result in captures after the IPSEC encryption.

In the mean time, I've examined the logs from the IPSEC replication and compared them to logs generated when cloning the initial data set to the test environment.

In addition to the corrupt object messages, I've noticed a lot of these messages in the IPSEC replication logs that are not present in the logs when cloning over the local network:

Socket recv() conn closed? 54.86.48.186:43237
SocketException: remote: 54.86.48.186:43237 error: 9001 socket exception [CLOSED] server [54.86.48.186:43237]

These messages appear to suggest that the connection between the two instances are being interrupted at some point. Is it possible that these connection errors are causing the corrupt object issues?

Could the corruption be due to messages between the two servers getting truncated unexpectedly?

Comment by David Hows [ 09/May/14 ]

Hi Brian,

MongoDB shouldn't have any direct interactions with the IPSEC itself (that should be handled lower in the stack) so I cant see how this would effect things directly.

With this in mind would you be able to gather a few datapoints for us to investigate this better. Next time you run the resync over your IPSEC tunnel can you please perform some packet captures at either end outside the tunnel. I'd like to see

  1. What the from side sends
  2. What the to side receives
  3. What the syncing member reports

Hopefully between those three pieces of data we can work out exactly what is going on and from there what is going wrong.

Thanks,
David

Comment by Brian Bernstein [ 21/Apr/14 ]

Log file of secondary instance during initial sync.

Comment by Brian Bernstein [ 21/Apr/14 ]

I've repeated my replication testing across the link and the number of corrupted documents varies during each attempt.

Out of a collection size of 113501, the number of corrupted documents after replication will vary between 8 to 13 documents.

I had set the log level to 4 in one of the replication attempts. I'll attach the log file to this issue.

Comment by Brian Bernstein [ 19/Apr/14 ]

On the second attempt to perform the initial sync over the IPSec link, the logs contained different ObjectId's compared to the first attempt. However, the number of corrupted documents appeared to be the same-- the respective counts for the primary and secondary instances of that collection did not change.

I performed a simple test of the IPSec link to ensure it was working properly. I transferred a 100MB file across the IPSec link with netcat and compared the md5sum at both ends. The test worked successfully.

Comment by Eliot Horowitz (Inactive) [ 19/Apr/14 ]

Is it always the same documents or are they different?
Have you tested the ipsec link in general to make sure its clean?

Generated at Thu Feb 08 03:32:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.