[SERVER-64304] Using --recoverFromOplogAsStandalone can result in index builds crashing the server Created: 07/Mar/22  Updated: 29/Oct/23  Resolved: 09/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.4.14, 6.0.0-rc0, 5.0.7

Type: Bug Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-54968 recoverFromOplogAsStandalone doesn't ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0, v4.4
Steps To Reproduce:

See Matt's patch build.

Sprint: Execution Team 2022-03-21
Participants:
Linked BF Score: 146

 Description   

When starting mongod with --recoverFromOplogAsStandalone, unfinished index builds don't get resumed:

STORAGE  21005   [initandlisten] "Not restarting unfinished index builds because we are in standalone mode"

However, if we come across an oplog entry to start an index build, it will begin building the index.

Once the oplog is replayed, the server enters into a read-only mode:

REPL     21558   [initandlisten] "Setting mongod to readOnly mode as a result of specifying 'recoverFromOplogAsStandalone'"

If the index build was unfinished at this point, it will result in a crash:

STORAGE  20649   [IndexBuildsCoordinatorMongod-0] "Index build: failed","attr":{"buildUUID":{"uuid":{"$uuid":"5be501ba-c27f-4a31-83fa-b865a3e58ca8"}},"collectionUUID":{"uuid":{"$uuid":"8db4911e-8b9e-4904-8496-3b407c876582"}},"namespace":"test.a","error":{"code":20,"codeName":"IllegalOperation","errmsg":"Cannot execute a write operation in read-only mode"}}
ASSERT   23081   [IndexBuildsCoordinatorMongod-0] "Invariant failure","attr":{"expr":"status.isA<ErrorCategory::Interruption>() || status.isA<ErrorCategory::ShutdownError>() || status.code() == ErrorCodes::CannotCreateIndex","msg":"Unexpected error code during index build cleanup: IllegalOperation: Cannot execute a write operation in read-only mode","file":"src/mongo/db/index_builds_coordinator.cpp","line":2210}

 



 Comments   
Comment by Githook User [ 10/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Prevent a server crash due to index builds started when using the --recoverFromOplogAsStandalone startup parameter

(cherry picked from commit d563fc234fedf423e55f99bc840ba8188b523603)
Branch: v4.4
https://github.com/mongodb/mongo/commit/1ed0b6074bd5bf6670c11404cc71460ee92bcc05

Comment by Githook User [ 10/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Prevent a server crash due to index builds started when using the --recoverFromOplogAsStandalone startup parameter
Branch: v5.0
https://github.com/mongodb/mongo/commit/c084c1aafe116e90afcc1e6db35fd574f387311f

Comment by Githook User [ 10/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Prevent a server crash due to index builds started when using the --recoverFromOplogAsStandalone startup parameter

(cherry picked from commit c83c80d2e58040c373a9f29be0c3f7b6595ea7c6)
Branch: v5.0
https://github.com/10gen/mongo-enterprise-modules/commit/c2151c2331ced1404bd52972441b0e8aeecb0d72

Comment by Githook User [ 09/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Prevent a server crash due to index builds started when using the --recoverFromOplogAsStandalone startup parameter
Branch: master
https://github.com/mongodb/mongo/commit/d563fc234fedf423e55f99bc840ba8188b523603

Comment by Githook User [ 09/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Resolve TODOs for selective_backup_restore_pit.js
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/9273133378e922d49a719436fc672b1bbdf018d5

Comment by Githook User [ 09/Mar/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-64304 Prevent a server crash due to index builds started when using the --recoverFromOplogAsStandalone startup parameter
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/c83c80d2e58040c373a9f29be0c3f7b6595ea7c6

Generated at Thu Feb 08 06:00:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.