[SERVER-30983] Investigate background index behavior with causal consistency and secondary read preference Created: 07/Sep/17 Updated: 27/Oct/23 Resolved: 08/Dec/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.6.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Jack Mulrow | Assignee: | [DO NOT USE] Backlog - Sharding Team |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | sharding36-passthrough-testing | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Sharding
|
| Backwards Compatibility: | Fully Compatible |
| Operating System: | ALL |
| Sprint: | Sharding 2017-10-02, Sharding 2017-10-23 |
| Participants: |
| Description |
|
Possible bug with secondary read preference and causal consistency on. The reindex_background.js workload was failing with an "IndexNotFound" error somewhat frequently in the concurrency_sharded_causal_consistency(_and_balancer) suites once I turned secondary read preference on. Example stack trace:
|
| Comments |
| Comment by Spencer Brody (Inactive) [ 05/Dec/17 ] |
|
Agreed with Will's summary. I don't think there's anything replication can do about this at the moment. Sounds like this likely depends on PM-253 (index builds with two phase commit), which is on the storage backlog. Sending back to Sharding. |
| Comment by William Schultz (Inactive) [ 05/Dec/17 ] |
|
misha.tyulenev, to your comment, this sort of seems like a fundamental incompatibility between replicated indexes and causal consistency, at least as the system stands today. As far as I understand, the only operation about an index build that we actually log in the oplog is the createIndex op. If the index creation is synchronous i.e. background=false, then the client can explicitly wait until the build has finished, and then, presumably, use the operationTime of the index build operation to read after. If the build is asynchronous i.e. background=true, then the only information the client can receive about that index's lifecycle is the optime of the createIndex operation. The actual operations carried out internally to build the physical index are not explicitly "replicated", since they are not logged in the oplog. Additionally, there would be no way, from the oplog's perspective, to determine when a background index build "finished". So, it seems like there may be no existing way for a client to check for the completion of a background index build simply by utilizing the causal consistency machinery. The most they could do is a read that guarantees it happened after the start of the index build, since that seems to be the only operation time that actually exists in relation to the lifetime of a replicated index. Let me know if I am misunderstanding any of the fundamental concepts here. There doesn't really seem to be a fix for this specific issue unless the behavior of replicated index builds are significantly changed. |
| Comment by Benety Goh [ 31/Oct/17 ] |
|
Assigning to william.schultz to get a minimal reproduction of the issue. |
| Comment by Misha Tyulenev [ 27/Oct/17 ] |
|
max.hirschhorn until this issue is resolved all tests that involve background index building are not causally consistent. |
| Comment by Misha Tyulenev [ 27/Oct/17 ] |
|
The issue is caused by how ensureIndex( {background: true}) command is replicated. |
| Comment by Esha Maharishi (Inactive) [ 20/Oct/17 ] |
|
Could this be related to BF-6752, where there is a race between migrations and createIndexes that causes an index not to be created on a recipient shard? |
| Comment by Max Hirschhorn [ 09/Oct/17 ] |
|
misha.tyulenev, the failures in the reindex_background.js FSM workload are not caused by the issue described in |