[SERVER-33093] catalog::openCatalog must rebuild all indexes for a collection in one pass Created: 02/Feb/18  Updated: 29/Oct/23  Resolved: 13/Feb/18

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: None
Fix Version/s: 3.7.3

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Kyle Suarez
Resolution: Fixed Votes: 0
Labels: rollback-functional
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-29213 Have KVWiredTigerEngine implement Sto... Closed
Related
is related to SERVER-33322 reconcileCatalogAndIdents() should al... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Repl 2018-02-12, Repl 2018-02-26
Participants:

 Description   

KVStorageEngine::reconcileCatalogAndIdents returns a list of (collection name, index name) pairs that need to be rebuilt. If a collection requires rebuilding multiple indexes, they will be separate entries in this list.

This API suggests that rebuilding indexes one by one would work, but that's unfortunately not true.[1]

There are two obvious ways to correct catalog::openCatalog rebuilding indexes one by one.

  1. Complete the TODO to enable building an index without the collection object.
  2. Have catalog::openCatalog group all the indexes to rebuild per collection before passing them to rebuildIndexesOnCollection.

[1] More completely: creating the collection object will cascade and create all of the in-memory index objects for that collection. The WiredTigerIndex constructor will fail when searching for the underlying table. rebuildIndexesOnCollection avoids this situation by first dropping all indexes that are to be rebuilt from the storage engine's implementation of the collection catalog. Thus when the Collection object is constructed, it does not observe the indexes to be rebuilt. Thus this bug only manifests when there is more than one missing index to rebuild on a collection.



 Comments   
Comment by Siyuan Zhou [ 14/Feb/18 ]

Since the commit got reverted, no BF was filed according to the BF procedure. Tracking the issue in SERVER-33322 is sufficient. Thanks for letting me know! CC the current Build Baron, adam.martin.

Comment by Kyle Suarez [ 14/Feb/18 ]

Filed SERVER-33322 for the index building fix.

Comment by Kyle Suarez [ 14/Feb/18 ]

daniel.gottlieb, looks like reconcileCatalogAndIdents() shouldn't include unready indexes in the list of indexes to rebuild when the --noIndexBuildRetry option is set:

[js_test:index_no_retry] 2018-02-13T22:06:10.241+0000 2018-02-13T22:06:10.240+0000 E QUERY    [thread1] Error: did not throw exception: index {a: 1} was rebuilt in spite of --noIndexBuildRetry :
[js_test:index_no_retry] 2018-02-13T22:06:10.241+0000 doassert@src/mongo/shell/assert.js:18:14
[js_test:index_no_retry] 2018-02-13T22:06:10.242+0000 assert.throws@src/mongo/shell/assert.js:365:13
[js_test:index_no_retry] 2018-02-13T22:06:10.242+0000 @jstests/noPassthrough/index_no_retry.js:56:1
[js_test:index_no_retry] 2018-02-13T22:06:10.242+0000 @jstests/noPassthrough/index_no_retry.js:7:2
[js_test:index_no_retry] 2018-02-13T22:06:10.242+0000 failed to load: jstests/noPassthrough/index_no_retry.js

If we want to fix this, should we just open a separate ticket to track the work, given that we won't revert the other commit (which is the main point of this ticket)?

Comment by Kyle Suarez [ 14/Feb/18 ]

I'm glad I committed them separately

siyuan.zhou, could you please link any build failures caused by that commit to this ticket?

Comment by Githook User [ 14/Feb/18 ]

Author:

{'email': 'siyuan.zhou@mongodb.com', 'name': 'Siyuan Zhou', 'username': 'visualzhou'}

Message: Revert "SERVER-33093 include unready indexes for rebuilding when reconciling catalog"

This reverts commit 91b0a8c599cae1a130f8450e37681b3a134b5dd6.
Branch: master
https://github.com/mongodb/mongo/commit/c4e444a62ce32eb37928396699fc7aaa0536226e

Comment by Githook User [ 13/Feb/18 ]

Author:

{'email': 'kyle.suarez@mongodb.com', 'name': 'Kyle Suarez', 'username': 'ksuarz'}

Message: SERVER-33093 include unready indexes for rebuilding when reconciling catalog
Branch: master
https://github.com/mongodb/mongo/commit/91b0a8c599cae1a130f8450e37681b3a134b5dd6

Comment by Githook User [ 13/Feb/18 ]

Author:

{'email': 'kyle.suarez@mongodb.com', 'name': 'Kyle Suarez', 'username': 'ksuarz'}

Message: SERVER-33093 rebuild all indexes in one pass in openCatalog()
Branch: master
https://github.com/mongodb/mongo/commit/17e104e78f5682d4caf7a1ee39788201fd619ec0

Generated at Thu Feb 08 04:32:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.