[SERVER-30952] Initial (re)sync never completes, stuck in a loop Created: 05/Sep/17 Updated: 07/Apr/23 Resolved: 05/Sep/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 3.4.7 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Robert Beekman | Assignee: | Kelsey Schubert |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Steps To Reproduce: |
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
We have a 3-node replicaset running v3.4.7, the primary is running 6 map/reduce jobs every minute or so and due to circumstances we had to re-sync one of the secondary nodes. However at one of the last steps of the re-sync we get the following error:
After this, MongoDB cleans up the files and starts the re-sync again, it's now basically stuck in a very big loop. This used to work fine with 2.x, when we re-synced quite a few times. I'm not sure what to do about this, we can't stop the map/reduce jobs for the duration of the resync as it takes about 8 hours to get to this point. |
| Comments |
| Comment by Spencer Brody (Inactive) [ 07/Sep/17 ] | ||||
|
Hi thijs, For now I encourage you to try out Ramon's workaround, it seems the lowest impact way to work around this issue, although unfortunately it only works for avoiding the renames in mapReduce $out, it doesn't work for aggregations with $out. If you'd like help figuring out the issue with indexes that are preventing you from completing initial sync on 3.2.11, please let us know and we'd be happy to work with you to understand what's happening there. Sorry again for the difficulties this issue has caused you. -Spencer | ||||
| Comment by Ramon Fernandez Marina [ 07/Sep/17 ] | ||||
|
I think there's a better workaround than downgrading (which may have other issues). You can use output to a collection with an action. For example:
Using this approach avoids the implicit renameCollection operation used by the out option of mapReduce altogether and should help with the initial sync issue originally reported. Regards, | ||||
| Comment by Thijs Cadier [ 07/Sep/17 ] | ||||
|
Unfortunately we can't seem to sync a 3.2.11 node, the sync fails with an error message about missing indexes. We've started work on a solution to completely move away from using aggregations in our processing pipeline. I think this change was communicated extremely poorly. It was introduced in 3.2.12 with benign sounding description "renameCollection ‘c’ op should restart initial sync upon application". Also it's not even mentioned in the overview page here (https://docs.mongodb.com/manual/release-notes/3.2/), you have to click to the detailed list to even find it. There is no mention at all that this change would break initial syncs when you use big aggregations. This is a change with huge impact. I really don't understand that this change was made in a point release and didn't come with a huge warning in the change log. | ||||
| Comment by Daniel Pasette (Inactive) [ 06/Sep/17 ] | ||||
|
Glad you have an escape valve. Please look through the caveats Spencer called out and remember that the version must be 3.2.11 or lower. | ||||
| Comment by Thijs Cadier [ 06/Sep/17 ] | ||||
|
Thanks for the thoughtful reply Spencer. Great news that this can be mitigated by running 3.2 We were under the impression that this issue was also present in 3.2. We can still run 3.2 since we haven't performed the auth upgrade yet. We are trying the downgrade approach now, will let you know how it goes. | ||||
| Comment by Spencer Brody (Inactive) [ 06/Sep/17 ] | ||||
|
Hi thijs and robert@80beans.com, Until 3.6 is released, however, I understand you are in a tricky situation. The best thing to do is one of the workarounds that Thomas describes - either cease running the aggregations and mapReduces causing the problem for the duration of the initial sync, or rework the app to not use $out to have the agg or m/r write directly to the output collection, but instead get the results streamed back to your app then have your app manually write the data out to the output collection. If neither of those are an option, you can downgrade the initially syncing node to 3.2.11 or wait for I hope this clarifies the issue and the steps to work around it. I recognize that this is a difficult bug to work around and can cause real operational hurdles for you, and for that I sincerely apologize on behalf of MongoDB. We are working hard to get this fixed for 3.6. -Spencer | ||||
| Comment by Thijs Cadier [ 06/Sep/17 ] | ||||
|
Hi, We're in a very tight spot at the moment because of this issue:
Basically we need a fix for this otherwise we have to do an extremely hasty project to take this entire system out of commission To me it seems like this issue is not a duplicate of Why is a fix for this issue that causes a real risk of data-loss not given more priority? Thijs | ||||
| Comment by Kelsey Schubert [ 05/Sep/17 ] | ||||
|
Unfortunately, there is no simple workaround for this issue. There are two possible workarounds, the first you've already identified and ruled out:
Kind regards, | ||||
| Comment by Robert Beekman [ 05/Sep/17 ] | ||||
|
Will do, but that issue looks unresolved. In the meantime we're down one replicaset member. Is there any workaround to get an initial sync to complete? | ||||
| Comment by Kelsey Schubert [ 05/Sep/17 ] | ||||
|
Thanks for the report, this issue is being tracked under Kind regards, | ||||
| Comment by Robert Beekman [ 05/Sep/17 ] | ||||
|
Additional information: The map/reduce jobs are preceded by an aggregation that also creates a temp collection, this because the aggregation is a lot faster then the map/reduce jobs. |