[SERVER-44343] Make 'reIndex' a standalone-only command Created: 31/Oct/19 Updated: 29/Oct/23 Resolved: 16/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Storage |
| Affects Version/s: | 4.2.1, 4.3.1 |
| Fix Version/s: | 4.7.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Gregory Wlodarek | Assignee: | Samyukta Lanka |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
|||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Minor Change | |||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | |||||||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | Run the following script using --suites=replica_sets.
|
|||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Repl 2019-12-02, Repl 2019-12-16, Repl 2019-12-30, Repl 2020-01-13, Repl 2020-03-09, Repl 2020-03-23, Repl 2020-04-06, Repl 2020-04-20 | |||||||||||||||||||||||||||||||||||||||||||||
| Participants: | ||||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 14 | |||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Trying to run the 'reIndex' command on a secondary while having unfinished prepared transactions will hit an invariant. After dropping the index that will be re-indexed, we hit an invariant when doing a collection scan to insert all the documents in the collection for that index.
ldeng recommended to try set the 'canIgnorePrepareConflicts' flag to true for the 'reIndex' command but that ended up hitting another invariant while trying to drop all the indexes.
We should consider disallowing the 'reIndex' command from running on secondaries and removing it in a future release. |
| Comments |
| Comment by Githook User [ 16/Apr/20 ] | |||||||||||||||||||||||||
|
Author: {'name': 'Samy Lanka', 'email': 'samy.lanka@mongodb.com', 'username': 'lankas'}Message: | |||||||||||||||||||||||||
| Comment by Githook User [ 13/Apr/20 ] | |||||||||||||||||||||||||
|
Author: {'name': 'Samy Lanka', 'email': 'samy.lanka@mongodb.com', 'username': 'lankas'}Message: Revert " This reverts commit 2ad12521309aeccc6f712f4ccb532f113255d186. | |||||||||||||||||||||||||
| Comment by Githook User [ 09/Apr/20 ] | |||||||||||||||||||||||||
|
Author: {'name': 'Samy Lanka', 'email': 'samy.lanka@mongodb.com', 'username': 'lankas'}Message: | |||||||||||||||||||||||||
| Comment by Eric Milkie [ 18/Jan/20 ] | |||||||||||||||||||||||||
|
The reason why I asked is because I am unsure of the current use case for reIndex. Why would someone want to rebuild an index only on their primary node – I would think that the primary node would be the one node state that a user would least likely want to run the reIndex command. | |||||||||||||||||||||||||
| Comment by Judah Schvimer [ 16/Jan/20 ] | |||||||||||||||||||||||||
|
This is not suggesting adding functionality to 'reIndex', just removing the ability to run it on secondaries. My understanding is 'reIndex' is currently supported on both primaries and secondaries, so after this it would only be supported on primaries. If a user wants to reindex an index on a secondary, they will need to restart the node as a standalone. Alternatively we could make it a standalone-only command, which evin.roesle would have to weigh in on. That would be even easier than making it primary-only. | |||||||||||||||||||||||||
| Comment by Eric Milkie [ 16/Jan/20 ] | |||||||||||||||||||||||||
|
Would this new primary-only command be replicated? | |||||||||||||||||||||||||
| Comment by Judah Schvimer [ 10/Dec/19 ] | |||||||||||||||||||||||||
|
It wouldn't be, I didn't understand how kIgnoreConflictsAllowWrites worked and why it was safe to use in other index building scenarios. louis.williams explained to me how it works and why it is unsafe here but safe in other scenarios. This was suggested in a repl team meeting and I wanted to try it, to gain understanding. Together we thought of another solution, other than banning reIndex on secondaries. reIndex could uassert and fail whenever it hit a prepare conflict. This would mean that the user would lose their indexes since reIndex is not transactional, but that is a known problem with reIndex. This would exacerbate it though:
I think banning reIndex on secondaries and encouraging users to restart secondaries as standalones and call reIndex on the standalone would be preferable. evin.roesle, do you know if banning reIndex on secondaries would be an acceptable solution to this invariant? | |||||||||||||||||||||||||
| Comment by Eric Milkie [ 10/Dec/19 ] | |||||||||||||||||||||||||
|
Why would ignoring prepared documents be safe for index builds on secondaries? | |||||||||||||||||||||||||
| Comment by Judah Schvimer [ 10/Dec/19 ] | |||||||||||||||||||||||||
|
I added the following diff in the reIndex code, and while the test no longer invariants, it gets a dbhash mismatch saying the _id:1 document is missing on the secondary.
|