[SERVER-37498] remove slowBackgroundIndexBuild failpoint, which skips documents when building index, leading to missing index entries Created: 05/Oct/18  Updated: 29/Oct/23  Resolved: 11/Jan/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.1.7

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-38527 Reduce the number of supports*() func... Closed
is depended on by SERVER-38820 Unjournaled writes should be durable ... Closed
Related
is related to SERVER-33377 MultiIndexBlock builder can incorrect... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

python buildscripts/resmoke.py --suites=no_server repro_server37498.js

repro_server37498.js

(function() {
    "use strict";
 
    const conn = MongoRunner.runMongod();
    assert.neq(null, conn, "mongod was unable to start up");
 
    const db = conn.getDB("test");
 
    assert.commandWorked(
        db.adminCommand({configureFailPoint: "slowBackgroundIndexBuild", mode: "alwaysOn"}));
 
    assert.commandWorked(db.mycoll.insert({a: 0}));
    assert.commandWorked(db.mycoll.createIndex({a: 1}, {background: true}));
 
    assert.commandWorked(
        db.adminCommand({configureFailPoint: "slowBackgroundIndexBuild", mode: "off"}));
 
    MongoRunner.stopMongod(conn);
})();

[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 Collection validation failed on host localhost:20020 with response: {
[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 	"ns" : "test.mycoll",
[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 	"nInvalidDocuments" : NumberLong(0),
[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 	"nrecords" : 1,
[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 	"nIndexes" : 2,
[js_test:repro_server37498] 2018-10-05T20:32:01.133-0500 	"keysPerIndex" : {
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		"test.mycoll.$_id_" : 1,
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		"test.mycoll.$a_1" : 0
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 	},
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 	"indexDetails" : {
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		"test.mycoll.$_id_" : {
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 			"valid" : false
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		},
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		"test.mycoll.$a_1" : {
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 			"valid" : false
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 		}
[js_test:repro_server37498] 2018-10-05T20:32:01.134-0500 	},
[js_test:repro_server37498] 2018-10-05T20:32:01.135-0500 	"valid" : false,
[js_test:repro_server37498] 2018-10-05T20:32:01.135-0500 	"warnings" : [ ],
[js_test:repro_server37498] 2018-10-05T20:32:01.135-0500 	"errors" : [
[js_test:repro_server37498] 2018-10-05T20:32:01.135-0500 		"one or more indexes contain invalid index entries."
[js_test:repro_server37498] 2018-10-05T20:32:01.135-0500 	],
[js_test:repro_server37498] 2018-10-05T20:32:01.137-0500 	"advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.",
[js_test:repro_server37498] 2018-10-05T20:32:01.137-0500 	"ok" : 1
[js_test:repro_server37498] 2018-10-05T20:32:01.137-0500 }

Sprint: Storage NYC 2018-12-31, Storage NYC 2019-01-14
Participants:

 Description   

The issue is that the PlanExecutor::getNextSnapshotted() is continually called while the slowBackgroundIndexBuild failpoint is set without ever calling MultiIndexBlockImpl::insert(). It may be sufficient to set retries=1 before the continue, but I'd be curious if the slowBackgroundIndexBuild failpoint is still necessary after the introduction of the hangBeforeIndexBuildOf and hangAfterIndexBuildOf failpoints as part of SERVER-26387.

src/mongo/db/catalog/index_create_impl.cpp

auto exec =
    InternalPlanner::collectionScan(_opCtx, _collection->ns().ns(), _collection, yieldPolicy);
 
Snapshotted<BSONObj> objToIndex;
RecordId loc;
PlanExecutor::ExecState state;
int retries = 0;  // non-zero when retrying our last document.
while (retries ||
        (PlanExecutor::ADVANCED == (state = exec->getNextSnapshotted(&objToIndex, &loc))) ||
        MONGO_FAIL_POINT(hangAfterStartingIndexBuild)) {
    try {
        if (_allowInterruption)
            _opCtx->checkForInterrupt();
 
        if (!(retries || PlanExecutor::ADVANCED == state) ||
            MONGO_FAIL_POINT(slowBackgroundIndexBuild)) {
            log() << "Hanging index build due to failpoint";
            invariant(_allowInterruption);
            sleepmillis(1000);
            continue;
        }



 Comments   
Comment by Githook User [ 23/Jan/19 ]

Author:

{'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek', 'username': 'GWlodarek'}

Message: SERVER-37498 Add requires_majority_read_concern flag for test
Branch: master
https://github.com/mongodb/mongo/commit/6ece5d5c8da8403781c641e35c7e8e1d96f6e10f

Comment by Githook User [ 11/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 Remove the slowBackgroundIndexBuild failpoint
Branch: master
https://github.com/mongodb/mongo/commit/df09162ac73affa30dc03077ae6812e8f99d257f

Comment by Githook User [ 11/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 Change jstests to not use the slowBackgroundIndexBuild failpoint
Branch: master
https://github.com/mongodb/mongo/commit/4a3fb2fe383f9f77ffbbef0a30eefdff57e9a631

Comment by Githook User [ 04/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-37498 remove slowBackgroundIndexBuild failpoint"

This reverts commit 46dd8e03cdef50a4d3b5714619d7a097c9a1578f.
Branch: master
https://github.com/mongodb/mongo/commit/0cfba57ca39fc2e96bb003f9d52659d436b3a768

Comment by Githook User [ 04/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-37498 Ensure writes are durable on the secondary"

This reverts commit ba7f1223210a95b18b14271e498fb18c5b5e7b96.
Branch: master
https://github.com/mongodb/mongo/commit/ff7e8811a933e0ebed74fc2615c6865a91114c62

Comment by Githook User [ 02/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 Ensure writes are durable on the secondary
Branch: master
https://github.com/mongodb/mongo/commit/ba7f1223210a95b18b14271e498fb18c5b5e7b96

Comment by Githook User [ 01/Jan/19 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 remove slowBackgroundIndexBuild failpoint
Branch: master
https://github.com/mongodb/mongo/commit/46dd8e03cdef50a4d3b5714619d7a097c9a1578f

Comment by Githook User [ 31/Dec/18 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-37498 remove slowBackgroundIndexBuild failpoint"

This reverts commit 4a0e4f7f0eeeda0b5a0ef6f1867d7d144de329c0.
Branch: master
https://github.com/mongodb/mongo/commit/d0115b010ff519681a6152616539e6d20862163e

Comment by Githook User [ 31/Dec/18 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 remove slowBackgroundIndexBuild failpoint
Branch: master
https://github.com/mongodb/mongo/commit/4a0e4f7f0eeeda0b5a0ef6f1867d7d144de329c0

Comment by Githook User [ 31/Dec/18 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: Revert "SERVER-37498 remove slowBackgroundIndexBuild failpoint"

This reverts commit cc1ac3b8e3ba813de5321d7b7837865feeedf356.
Branch: master
https://github.com/mongodb/mongo/commit/110d33066a44bd77536c5cc34a44ecf46dd3d31e

Comment by Githook User [ 31/Dec/18 ]

Author:

{'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'name': 'Gregory Wlodarek'}

Message: SERVER-37498 remove slowBackgroundIndexBuild failpoint
Branch: master
https://github.com/mongodb/mongo/commit/cc1ac3b8e3ba813de5321d7b7837865feeedf356

Generated at Thu Feb 08 04:46:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.