[SERVER-35658] session migration is too sensitive to replica set primary elections Created: 18/Jun/18 Updated: 29/Oct/23 Resolved: 24/Jul/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.6.5 |
| Fix Version/s: | 3.6.7, 4.0.2, 4.1.2 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Elan Kugelmass | Assignee: | Matthew Saltz (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
MongoD/MongoS 3.6.5 |
||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Operating System: | ALL | ||||
| Backport Requested: |
v4.0, v3.6
|
||||
| Sprint: | Sharding 2018-07-16, Sharding 2018-07-30 | ||||
| Participants: | |||||
| Description |
|
Retryable writes was added in v3.6, and the way the server remembers a write is written is by tagging the oplog entry for the write. During migration, the server will also need to transfer the oplog entries related to retryable write to the destination shard. It also has to make sure that the oplog entries it sends doesn't get rolled back. The current code is extra paranoid and can trigger the assert even though the oplog entries are majority committed.
Original Description: One collection will not migrate chunks due to change of term, but reports stale term
|
| Comments |
| Comment by Ramon Fernandez Marina [ 27/Aug/18 ] |
|
Thanks for the update epkugelmass, and glad to hear you were able to balance your cluster with 3.6.7. Regards, |
| Comment by Elan Kugelmass [ 26/Aug/18 ] |
|
Confirmed we're able to balance our cluster once we upgrade to 3.6.7. Thanks for the fix! |
| Comment by Githook User [ 06/Aug/18 ] |
|
Author: {'name': 'Matthew Saltz', 'email': 'matthew.saltz@mongodb.com', 'username': 'saltzm'}Message: (cherry picked from commit 9175c4deba82dc35606d14428d1bf0d8b43d7a6c) |
| Comment by Githook User [ 24/Jul/18 ] |
|
Author: {'name': 'Matthew Saltz', 'email': 'matthew.saltz@mongodb.com'}Message: (cherry picked from commit 9175c4deba82dc35606d14428d1bf0d8b43d7a6c) |
| Comment by Githook User [ 24/Jul/18 ] |
|
Author: {'name': 'Matthew Saltz', 'email': 'matthew.saltz@mongodb.com'}Message: |
| Comment by Randolph Tan [ 09/Jul/18 ] |
|
epkugelmass yes, we're aiming to backport it to v4.0 and possibly to v3.6. Currently, the safest way to get around this is to either start a new operation that will target the shard having the issue for every session in config.transactions or kill all sessions in the config.transactions of the shard and wait for the documents to go away (remove is done asynchronously). |
| Comment by Elan Kugelmass [ 05/Jul/18 ] |
|
renctan do you anticipate introducing a patch to make the behavior less paranoid? In the interim, is there anything I can do to get my cluster balancing again? |
| Comment by Randolph Tan [ 05/Jul/18 ] |
|
Retryable writes was added in v3.6, and the way the server remembers a write is written is by tagging the oplog entry for the write. During migration, the server will also need to transfer the oplog entries related to retryable write to the destination shard. It also has to make sure that the oplog entries it sends doesn't get rolled back. The current change is extra paranoid and can trigger the assert even though the oplog entries are majority committed. |
| Comment by Esha Maharishi (Inactive) [ 05/Jul/18 ] |
|
epkugelmass, I spoke to renctan and he will take it over as he is familiar with this code. |
| Comment by Elan Kugelmass [ 03/Jul/18 ] |
|
esha.maharishi renctan Can I propose raising the priority on this ticket to critical? The bug appears to be recurring, has no known workaround, and (due to the fact that the cluster can't balance) severely impacts the usability and value of a sharded cluster. |
| Comment by Elan Kugelmass [ 28/Jun/18 ] |
|
The problem has reappeared. SHARDING [conn106] Chunk move failed :: caused by :: CommandFailed: commit clone failed :: caused by :: Location40650: detected change of term from 231 to 232 |
| Comment by Elan Kugelmass [ 20/Jun/18 ] |
|
Scrolling through the mongod log, we also see "inverted" term error messages like exception: commit clone failed :: caused by :: Location40650: detected change of term from 231 to 223. After several days of the failures described here and in the initial description, the collection began successfully balancing and we haven't seen any problems since. |
| Comment by Esha Maharishi (Inactive) [ 18/Jun/18 ] |
|
It seems like that error message came from new code added for retryable writes... CC renctan |