[SERVER-55366] inject_tenant_prefix.js should not modify the original command on retrying retryable writes Created: 19/Mar/21  Updated: 29/Oct/23  Resolved: 01/Apr/21

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.9.0-rc1, 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Cheahuychou Mao Assignee: Pavithra Vetriselvan
Resolution: Fixed Votes: 0
Labels: pm-1791_non-cloud-blocking
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-52713 [testing] Add stepdown/kill/terminate... Closed
Problem/Incident
causes SERVER-55364 Investigate DuplicateKey error for cr... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.9
Sprint: Repl 2021-04-05
Participants:

 Description   

From the EVG patch builds for SERVER-52713, crud operations run as retryable writes appear to not always work correctly when there are background migrations.

[ContinuousTenantMigration:job7] Starting tenant migration: {'donor': 'rs0', 'recipient': 'rs1', 'migration_id': UUID('db9aed32-0810-499b-8e1a-a81500caa62d'), 'tenant_id': 'tenantMigrationTenantId', 'read_preference': {'mode': 'secondary'}}.
[ContinuousTenantMigration:job7] Starting tenant migration on donor primary on port 21752 of replica set 'rs0'.
[j7:rs0:n2] | 2021-03-19T03:28:35.068+00:00 I  TENANT_M 5093800 [TenantMigrationDonorService-0] "Tenant migration starting to block writes","attr":{"tenantId":"tenantMigrationTenantId"}
[js_test:insert1] [jsTest] ----
[js_test:insert1] [jsTest] Got TenantMigrationAborted for command against database "tenantMigrationTenantId_test" after trying 1 times, retrying the command: {
[js_test:insert1] [jsTest] 	"n" : 500,
[js_test:insert1] [jsTest] 	"electionId" : ObjectId("7fffffff0000000000000002"),
[js_test:insert1] [jsTest] 	"opTime" : {
[js_test:insert1] [jsTest] 		"ts" : Timestamp(1616124515, 1001),
[js_test:insert1] [jsTest] 		"t" : NumberLong(2)
[js_test:insert1] [jsTest] 	},
[js_test:insert1] [jsTest] 	"ok" : 1,
[js_test:insert1] [jsTest] 	"$clusterTime" : {
[js_test:insert1] [jsTest] 		"clusterTime" : Timestamp(1616124536, 1),
[js_test:insert1] [jsTest] 		"signature" : {
[js_test:insert1] [jsTest] 			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
[js_test:insert1] [jsTest] 			"keyId" : NumberLong(0)
[js_test:insert1] [jsTest] 		}
[js_test:insert1] [jsTest] 	},
[js_test:insert1] [jsTest] 	"operationTime" : Timestamp(1616124515, 1001),
[js_test:insert1] [jsTest] 	"truncatedWriteErrors" : [
[js_test:insert1] [jsTest] 		{
[js_test:insert1] [jsTest] 			"index" : 500,
[js_test:insert1] [jsTest] 			"code" : 325,
[js_test:insert1] [jsTest] 			"errmsg" : "Tenant migration aborted"
[js_test:insert1] [jsTest] 		}
[js_test:insert1] [jsTest] 	]
[js_test:insert1] [jsTest] }
[js_test:insert1] [jsTest] ----
[js_test:insert1] 
[ContinuousTenantMigration:job7] Tenant migration with donor primary on port 21752 of replica set 'rs0' has aborted: {'state': 'aborted', 'abortReason': {'code': 1, 'codeName': 'InternalError', 'errmsg': 'simulate a tenant migration error'}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1616124536, 1), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}, 'operationTime': Timestamp(1616124536, 1)}
[ContinuousTenantMigration:job7] Forgetting tenant migration: {'donor': 'rs0', 'recipient': 'rs1', 'migration_id': UUID('db9aed32-0810-499b-8e1a-a81500caa62d'), 'tenant_id': 'tenantMigrationTenantId', 'read_preference': {'mode': 'secondary'}}.
[ContinuousTenantMigration:job7] Forgetting tenant migration on donor primary on port 21752 of replica set 'rs0'.
[ContinuousTenantMigration:job7] Starting tenant migration: {'donor': 'rs0', 'recipient': 'rs1', 'migration_id': UUID('4169223b-5285-4ef0-a993-2fb60bb1a586'), 'tenant_id': 'tenantMigrationTenantId', 'read_preference': {'mode': 'secondary'}}.
[ContinuousTenantMigration:job7] Starting tenant migration on donor primary on port 21752 of replica set 'rs0'.
[js_test:insert1] New session started with sessionID: {  "id" : UUID("2bfb0351-54ac-4a57-ad1d-aa5bc5c1ef91") } and options: {  "readConcern" : {  "level" : "majority" },  "readPreference" : {  "mode" : "primary" },  "retryWrites" : true }
[js_test:insert1] assert: [100000] != [99500] are not equal : bad count
[js_test:insert1] doassert@src/mongo/shell/assert.js:20:14
[js_test:insert1] assert.eq@src/mongo/shell/assert.js:179:9
[js_test:insert1] @jstests/core/insert1.js:53:1



 Comments   
Comment by Githook User [ 08/Apr/21 ]

Author:

{'name': 'Pavi Vetriselvan', 'email': 'pavithra.vetriselvan@mongodb.com', 'username': 'pvselvan'}

Message: SERVER-55366 inject_tenant_prefix should always retry entire batch of retryable write operations

(cherry picked from commit 6a1453c287b0befcf73dc3f339af997c185b81e2)
Branch: v4.9
https://github.com/mongodb/mongo/commit/c8720273fde8f31f1cfbc61c143683a24a6f121f

Comment by Githook User [ 01/Apr/21 ]

Author:

{'name': 'Pavi Vetriselvan', 'email': 'pavithra.vetriselvan@mongodb.com', 'username': 'pvselvan'}

Message: SERVER-55366 inject_tenant_prefix should always retry entire batch of retryable write operations
Branch: master
https://github.com/mongodb/mongo/commit/6a1453c287b0befcf73dc3f339af997c185b81e2

Generated at Thu Feb 08 05:36:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.