Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-12957

Stopping a secondary during an index build can leave a non-usable index which is never reported

XMLWordPrintableJSON

    • ALL
    • Hide

      Create a replica set
      Have enough data in the set to make the index build take 10s of seconds
      Build an index on the primary
      Kill -5 the secondary when it is over 1/2 complete (follow the logs and look for something like the line below)

      2014-02-28T13:26:41.002+1100 [repl index builder 0]             Index Build(background): 775600/1358322 57%

      Start the secondary (it will not rebuild the index)
      The index is shown to exist in system.indexes but is not usable for queries nor does it show in validate()

      Show
      Create a replica set Have enough data in the set to make the index build take 10s of seconds Build an index on the primary Kill -5 the secondary when it is over 1/2 complete (follow the logs and look for something like the line below) 2014-02-28T13:26:41.002+1100 [repl index builder 0] Index Build(background): 775600/1358322 57% Start the secondary (it will not rebuild the index) The index is shown to exist in system.indexes but is not usable for queries nor does it show in validate()

      Similar to SERVER-12956, but on 2.6.0RC1 git rev 52fe0d21959e32a5bdbecdc62057db386e4e029c

      When a secondary is stopped during the build of an index it will not resume and complete the index build and the index will not be available for use.
      The last line of the index build was:

      2014-02-28T13:26:41.002+1100 [repl index builder 0]             Index Build(background): 775600/1358322 57%

      When running a query on that secondary i see the following:

      db.test.find({x:-2}).explain(true);
{
	"cursor" : "BasicCursor",
	"isMultiKey" : false,
	"n" : 1,
	"nscannedObjects" : 1358323,
	"nscanned" : 1358323,
	"nscannedObjectsAllPlans" : 1358323,
	"nscannedAllPlans" : 1358323,
	"scanAndOrder" : false,
	"indexOnly" : false,
	"nYields" : 0,
	"nChunkSkips" : 0,
	"millis" : 570,
	"indexBounds" : {

	},
	"allPlans" : [
		{
			"cursor" : "BasicCursor",
			"n" : 1,
			"nscannedObjects" : 1358323,
			"nscanned" : 1358323,
			"indexBounds" : {

			}
		}
	],
	"server" : "Pixl.local:27018"
}

      Validate does not test the index:

      ....
	"objectsFound" : 1358323,
	"invalidObjects" : 0,
	"bytesWithHeaders" : 347730432,
	"bytesWithoutHeaders" : 325997264,
	"deletedCount" : 15,
	"deletedSize" : 113161136,
	"nIndexes" : 1,
	"keysPerIndex" : {
		"test.test.$_id_" : 1358323
	},
	"valid" : true,
	"errors" : [ ],
	"ok" : 1
}

      But system.indexes shows it to exist:

      db.system.indexes.find();
{ "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "test.test" }
{ "v" : 1, "key" : { "x" : 1, "fruits" : 1, "transport" : 1 }, "name" : "x_1_fruits_1_transport_1", "ns" : "test.test", "background" : true }

      This looks to be a silent corruption and would only become evident when you examine why said index is not being used for query evaluation,

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            david.hows David Hows
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: