[SERVER-13654] InitialSync Replication over IPSEC link results in data corruption Created: 18/Apr/14 Updated: 10/Dec/14 Resolved: 10/Jun/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.4.10 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Brian Bernstein | Assignee: | David Hows |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 13.10 |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
When attempting to replicate a secondary member over the internet through an IPSEC link, the following messages are found in the logs in the secondary server during initial sync: [rsSync] Cloner: skipping corrupt object from testdb.testColl1 firstElement: _id: ObjectId('533440f68fde6a4ce2c4941e') These messages only occur when replicating over the internet, through the IPSEC link. When replicating locally, there is no issue. The initial sync succeeds, however the new secondary is missing records from the primary. Primary's testdb.testColl1.count(): Secondary's testdb.testColl1.count(): I have tested the reportedly corrupted objects by retrieving them from the primary, and can retrieve them fine. |
| Comments |
| Comment by Ramon Fernandez Marina [ 10/Jun/14 ] |
|
we haven't heard back from you for some time. Is there any further information you want to add to this ticket? To add to David's answer, if the IPSEC tunnel is truncating or dropping packets, then the BSON data is bound to get corrupted – at that point mongod refuses to insert it in the database. In the past I've run into issues with applications running over IPSEC tunnels where the solution was to lower the MTU, but this wasn't with MongoDB so your mileage may vary. At this stage it doesn't look like there's a bug in MongoDB, so I'm going to mark this issue as resolved. If you believe otherwise feel free to re-open the ticket and provide additional information. Regards, |
| Comment by David Hows [ 09/May/14 ] |
|
Hi Brian, I think you may be on the money. If the IPSEC tunnels are truncating packets or are truncating packets unexpectedly or silently dropping packets (silent as in causing them to appear as having never been sent) this could potentially cause problems. If the IPSEC tunnels are truncating the messages such that they are not "obviously" truncated (like not having missing sequence numbers for example) this could theoretically cause the issue. This was what i wished to investigate with the packet captures. Let me know how you go. Thanks, |
| Comment by Brian Bernstein [ 09/May/14 ] |
|
I am currently trying to figure out how to capture the network packets before the IPSEC encapsulation. The IPSEC tunnelling is occurring on the respective MongoDB's local hosts at the kernel level. Unfortunately, I think that simply running tcpdump on these instances will result in captures after the IPSEC encryption. In the mean time, I've examined the logs from the IPSEC replication and compared them to logs generated when cloning the initial data set to the test environment. In addition to the corrupt object messages, I've noticed a lot of these messages in the IPSEC replication logs that are not present in the logs when cloning over the local network: Socket recv() conn closed? 54.86.48.186:43237 These messages appear to suggest that the connection between the two instances are being interrupted at some point. Is it possible that these connection errors are causing the corrupt object issues? Could the corruption be due to messages between the two servers getting truncated unexpectedly? |
| Comment by David Hows [ 09/May/14 ] |
|
Hi Brian, MongoDB shouldn't have any direct interactions with the IPSEC itself (that should be handled lower in the stack) so I cant see how this would effect things directly. With this in mind would you be able to gather a few datapoints for us to investigate this better. Next time you run the resync over your IPSEC tunnel can you please perform some packet captures at either end outside the tunnel. I'd like to see
Hopefully between those three pieces of data we can work out exactly what is going on and from there what is going wrong. Thanks, |
| Comment by Brian Bernstein [ 21/Apr/14 ] |
|
Log file of secondary instance during initial sync. |
| Comment by Brian Bernstein [ 21/Apr/14 ] |
|
I've repeated my replication testing across the link and the number of corrupted documents varies during each attempt. Out of a collection size of 113501, the number of corrupted documents after replication will vary between 8 to 13 documents. I had set the log level to 4 in one of the replication attempts. I'll attach the log file to this issue. |
| Comment by Brian Bernstein [ 19/Apr/14 ] |
|
On the second attempt to perform the initial sync over the IPSec link, the logs contained different ObjectId's compared to the first attempt. However, the number of corrupted documents appeared to be the same-- the respective counts for the primary and secondary instances of that collection did not change. I performed a simple test of the IPSec link to ensure it was working properly. I transferred a 100MB file across the IPSec link with netcat and compared the md5sum at both ends. The test worked successfully. |
| Comment by Eliot Horowitz (Inactive) [ 19/Apr/14 ] |
|
Is it always the same documents or are they different? |