[SERVER-63865] Handle missing index idents during standalone startup recovery after unclean shutdown Created: 21/Feb/22  Updated: 21/Nov/23  Resolved: 31/May/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.1.0-rc0, 5.0.18, 6.0.7, 4.4.27

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: bkp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
is related to SERVER-67038 Handle missing index idents during st... Closed
is related to SERVER-62873 Startup repair does not handle orphan... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.0, v5.0, v4.4
Sprint: Execution Team 2022-05-30, Execution Team 2022-06-13
Participants:

 Description   

We may fail to start up in standalone mode after an unclean shutdown with unfinished index builds. This scenario specifically:

  • Replica set node crashes during an index build
  • During startup recovery as a replica set node, we drop the index table to rebuild the index, but the node crashes again. This may leave a state with an index in the catalog but a missing index table.
  • When starting the node in standalone mode, the following error occurs:

{"t":{"$date":"2022-01-19T14:30:53.262+00:00"},"s":"F",  "c":"-",        "id":23095,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28579,"error":"UnsupportedFormat: Unable to find metadata for table:index-0-5696206731032773165 Index: ... - version either too old or too new for this mongod.","file":"src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp","line":526}}

Currently, the only known remediation is "oplog surgery".

We need to ensure that unfinished indexes in standalone mode do not attempt to open the underlying table, and additionally ensure they can be dropped by the user in standalone mode.



 Comments   
Comment by Githook User [ 21/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)

GitOrigin-RevId: 9ea3bde687924531047c52d5041e5d3c7073ace1
Branch: v4.4-copybaratest
https://github.com/mongodb/mongo/commit/bfd77239305aaff1e6aa571de932151c69cab767

Comment by Githook User [ 21/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)

GitOrigin-RevId: 9ea3bde687924531047c52d5041e5d3c7073ace1
Branch: v4.4
https://github.com/mongodb/mongo/commit/0fd0cc32ce37fc2bc3937ffa390c719e84a4ab2b

Comment by Githook User [ 21/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)

GitOrigin-RevId: 9ea3bde687924531047c52d5041e5d3c7073ace1
Branch: v4.4-test
https://github.com/mongodb/mongo/commit/45cb15ef20312a766e3a7e87bbe0b1de3846ce58

Comment by Githook User [ 18/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)

GitOrigin-RevId: 9ea3bde687924531047c52d5041e5d3c7073ace1
Branch: v4.4
https://github.com/mongodb/mongo/commit/3f71e45682284ef638afe11a8a1e2986ec3e83d1

Comment by Githook User [ 17/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)

GitOrigin-RevId: 9ea3bde687924531047c52d5041e5d3c7073ace1
Branch: v4.4
https://github.com/mongodb/mongo/commit/1d34366abc29c87163ad86bcef3a4ebfb5343706

Comment by Githook User [ 08/May/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)
Branch: v5.0
https://github.com/mongodb/mongo/commit/67373f00d5cbfc2d674529611a61461a5815b694

Comment by Githook User [ 03/May/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery

(cherry picked from commit 0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d)
Branch: v6.0
https://github.com/mongodb/mongo/commit/b8a03580dee76fb55fa679e4c7c40dd3877d9140

Comment by Githook User [ 31/May/22 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery
Branch: master
https://github.com/mongodb/mongo/commit/0b5f72d6e251cb402aa3d8bb102cbba6b5734e8d

Comment by Githook User [ 31/May/22 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-63865 Handle missing index idents during standalone startup recovery
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/7a3d5a02f59a38b8307932dee74930cd110520a8

Generated at Thu Feb 08 05:58:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.