[SERVER-55355] Tenant migration recipient aborts the migration if it selected a stale sync source instead of selecting a new one Created: 19/Mar/21  Updated: 29/Oct/23  Resolved: 21/Apr/21

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.9.0-rc1, 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Cheahuychou Mao Assignee: Xuerui Fa
Resolution: Fixed Votes: 0
Labels: pm-1791_non-cloud-blocking
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-52713 [testing] Add stepdown/kill/terminate... Closed
Related
related to SERVER-56258 Complete TODO listed in SERVER-55355 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.9
Sprint: Repl 2021-04-19, Repl 2021-05-03
Participants:

 Description   

Based on the EVG patch builds for SERVER-52713, tenant migration recipient does not retry on InvalidSyncSource error as the error message indicates.

[j4:rs1:n1] | 2021-03-16T03:30:05.500+00:00 E  REPL     4881204 [TenantMigrationRecipientService-2] "Recipient migration service oplog fetcher failed","attr":{"tenantId":"tenantMigrationTenantId","migrationId":{"uuid":{"$uuid":"5b964868-1852-4fca-9c2f-e82f58cb3a4e"}},"error":{"code":119,"codeName":"InvalidSyncSource","errmsg":"Sync source's last applied OpTime { ts: Timestamp(1615865404, 3), t: 18 } is older than our last fetched OpTime { ts: Timestamp(1615865404, 5), t: -1 }. Choosing new sync source."}}
[j4:rs1:n1] | 2021-03-16T03:30:05.500+00:00 I  REPL     4881207 [TenantMigrationRecipientService-7] "Migration completed with both error and interrupt","attr":{"tenantId":"tenantMigrationTenantId","migrationId":{"uuid":{"$uuid":"5b964868-1852-4fca-9c2f-e82f58cb3a4e"}},"completionStatus":{"code":6,"codeName":"HostUnreachable","errmsg":"network error while attempting to run command 'listDatabases' on host 'localhost:21001' "},"interruptStatus":{"code":119,"codeName":"InvalidSyncSource","errmsg":"Sync source's last applied OpTime { ts: Timestamp(1615865404, 3), t: 18 } is older than our last fetched OpTime { ts: Timestamp(1615865404, 5), t: -1 }. Choosing new sync source."}}
[j4:rs1:n1] | 2021-03-16T03:30:05.500+00:00 I  REPL     4878501 [TenantMigrationRecipientService-7] "Tenant migration recipient instance: Data sync completed.","attr":{"tenantId":"tenantMigrationTenantId","migrationId":{"uuid":{"$uuid":"5b964868-1852-4fca-9c2f-e82f58cb3a4e"}},"error":{"code":119,"codeName":"InvalidSyncSource","errmsg":"Sync source's last applied OpTime { ts: Timestamp(1615865404, 3), t: 18 } is older than our last fetched OpTime { ts: Timestamp(1615865404, 5), t: -1 }. Choosing new sync source."}}
buildscripts.resmokelib.errors.ServerFailure: Tenant migration with donor primary on port 21002 of replica set 'rs0' has aborted due to an error: {'state': 'aborted', 'abortReason': {'code': 119, 'codeName': 'InvalidSyncSource', 'errmsg': "Tenant migration recipient command failed :: caused by :: Sync source's last applied OpTime { ts: Timestamp(1615865404, 3), t: 18 } is older than our last fetched OpTime { ts: Timestamp(1615865404, 5), t: -1 }. Choosing new sync source."}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1615865405, 930), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}, 'operationTime': Timestamp(1615865405, 930)}



 Comments   
Comment by Githook User [ 22/Apr/21 ]

Author:

{'name': 'XueruiFa', 'email': 'xuerui.fa@mongodb.com', 'username': 'XueruiFa'}

Message: SERVER-56258: Complete TODO from SERVER-55355

(cherry picked from commit fcf20de28b02f4f2a4e16addd87c43e722fb65e8)
Branch: v4.9
https://github.com/mongodb/mongo/commit/7c5eecd009a39e4f10dd4ad607cea642a0f10d83

Comment by Githook User [ 22/Apr/21 ]

Author:

{'name': 'XueruiFa', 'email': 'xuerui.fa@mongodb.com', 'username': 'XueruiFa'}

Message: SERVER-56258: Complete TODO from SERVER-55355
Branch: master
https://github.com/mongodb/mongo/commit/fcf20de28b02f4f2a4e16addd87c43e722fb65e8

Comment by Githook User [ 22/Apr/21 ]

Author:

{'name': 'XueruiFa', 'email': 'xuerui.fa@mongodb.com', 'username': 'XueruiFa'}

Message: SERVER-55355: Retry tenant migration on oplog fetcher errors

(cherry picked from commit e51091e12eec32b8fdb18ec56cf64928b1a61348)
Branch: v4.9
https://github.com/mongodb/mongo/commit/a48b6a726df973f9aa22b90c0f23f846c1fb985c

Comment by Githook User [ 21/Apr/21 ]

Author:

{'name': 'XueruiFa', 'email': 'xuerui.fa@mongodb.com', 'username': 'XueruiFa'}

Message: SERVER-55355: Retry tenant migration on oplog fetcher errors
Branch: master
https://github.com/mongodb/mongo/commit/e51091e12eec32b8fdb18ec56cf64928b1a61348

Generated at Thu Feb 08 05:36:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.