[SERVER-18822] Sharded clusters with WiredTiger primaries may lose writes during chunk migration Created: 04/Jun/15 Updated: 04/Aug/15 Resolved: 08/Jun/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency, Sharding, WiredTiger |
| Affects Version/s: | 3.0.0 |
| Fix Version/s: | 3.0.4 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Jeffrey Johnson | Assignee: | Andy Schwerin |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Steps To Reproduce: | The program I wrote to expose the bug is at: https://github.com/jeffhj11/MongoBug.git 1. Set up a cluster with three shards, each with WiredTiger as the storage engine. |
||||||||||||
| Sprint: | Sharding 5 06/26/16 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Issue Status as of Jun 11, 2015 ISSUE SUMMARY USER IMPACT
Under all of these conditions the target shard may lose writes to the chunk being migrated. It is not possible to determine if a cluster missed any writes because of this bug. None of the following deployments are affected by this issue:
WORKAROUND Users running WiredTiger-only sharded clusters may need to either add MMAPv1 nodes or replace WiredTiger nodes with MMAPv1 nodes, and make sure all remaining WiredTiger nodes are prevented from becoming primary as indicated above. Alternatively users can disable the balancer and cease manual chunk migrations, but note that this may lead to an unbalanced cluster and may have a negative impact on performance. The balancer can be disabled cluster-wide or on a per-collection basis. See the Documentation section below for more information. Users may upgrade WiredTiger nodes to 3.0.4-rc0 until 3.0.4 is released. AFFECTED VERSIONS FIX VERSION DOCUMENTATION Original descriptionThere appears to be a bug where data can go missing in MongoDB. The problem seems to occur when doing a large number of concurrent inserts into a sharded cluster while the balancer is running. In all instances where we lost data, wiredTiger was the storage engine and the shard key was effectively a random UUID, so that inserts were going to all shards. The test description and program shows the error with a Java application using a MongoDB database, but we also had the problem (less frequently) with a Python script. The bug seems to be some sort of concurrency/race condition problem. It is not guaranteed to happen on any one run, but we were able to replicate it fairly consistently. The number of documents missing range from 1-1000 during 400,000-800,000 inserts. Also note that we counted documents in two ways to determine that we had lost documents. The first was by running an aggregate to count the documents in an effort to avoid the way count() works when the balancer is running. We also waited for the balancer to finish and ran both a count() and an aggregate to ensure that documents are missing. System Configuration & Setup We tested with the following versions Cluster Setup We had the error occur in two different configurations, one with many servers and one mongod on each, and one with all mongo applications on one server. Single Server:
1 mongos, run with options:
Many Servers:
The replica sets were configured and primaries were elected prior to running the tests. Our application The Java application simply starts up a number of threads and each thread inserts a number of documents. There are options in the application to specify a message to more easily determine what data went missing. The document structure is simarly to:
The Tests
When that was completed, we would manually insert one document to start the balancer. We would watch the logs to ensure that the balancer was working. Once the balancer started, we would run our application again, so that multiple threads in the application were inserting documents into Mongo while the balancer was running. Common Themes and what we noticed The following seemed to be the common pieces that would cause MongoDB to lose data. Please keep in mind that this does not mean these are the actual causes, simply our observations and speculation. • We only lost data when WiredTiger was the storage engine, never with MMAP |
| Comments |
| Comment by Andy Schwerin [ 08/Jun/15 ] |
|
The root cause of this error was the same as the cause of The fix is to back port |
| Comment by Jeffrey Johnson [ 06/Jun/15 ] |
|
Thank you so much, I look forward to learning the details of the problem and when the fix will be available. |
| Comment by Ramon Fernandez Marina [ 05/Jun/15 ] |
|
jeffhj11, this is to let you know that we've identified the root cause of this issue and are working on a fix. We'll post updates on this ticket as they become available. Thanks, |
| Comment by Ramon Fernandez Marina [ 04/Jun/15 ] |
|
Thanks for the detailed report jeffhj11, we're investigating this issue and will let you know what we find. |