Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20764

create_index_gle.js intermittently fails, getLastError returns error "could not target full range of test.user; metadata not found"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.9
    • Component/s: None
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Sharding A (10/09/15)

      Description

      create_index_gle.js started intermittently failing since yesterday afternoon. The failure symptom is that a getLastError command returns a "could not target full range of test.user; metadata not found". I am unable to reproduce this failure on my desktop. A list follows of recent failures:

      Linux:

      SSL Ubuntu 14.04:

      SSL Amazon Linux:

      SSL RHEL 7.0:

      Excerpt from first failure in above list:

      [js_test:create_index_gle] 2015-10-01T18:54:39.800-0400 d20011| 2015-10-01T18:54:39.800-0400 D -        [conn13] User Assertion: 13435:not master and slaveOk=false
      [js_test:create_index_gle] 2015-10-01T18:54:39.801-0400 d20011| 2015-10-01T18:54:39.800-0400 D COMMAND  [conn13] assertion while executing command 'listDatabases' on database 'admin' with arguments '{ listDatabases: 1 }' and metadata '{ configsvr: { opTime: { ts: Timestamp 1443740079000|6, t: -1 } } }': 13435 not master and slaveOk=false
      [js_test:create_index_gle] 2015-10-01T18:54:39.801-0400 d20011| 2015-10-01T18:54:39.800-0400 I COMMAND  [conn13] command admin.$cmd command: listDatabases { listDatabases: 1 } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 exception: not master and slaveOk=false code:13435 numYields:0 reslen:125 locks:{} protocol:op_command 0ms
      [js_test:create_index_gle] 2015-10-01T18:54:39.803-0400 s20015| 2015-10-01T18:54:39.802-0400 I SHARDING [conn1] distributed lock with ts: 560db9afa3a189e276ebc8da' unlocked.
      [js_test:create_index_gle] 2015-10-01T18:54:39.803-0400 s20015| 2015-10-01T18:54:39.802-0400 W SHARDING [conn1] could not initialize targeter for index write op in collection test.user
      [js_test:create_index_gle] 2015-10-01T18:54:39.805-0400 s20015| 2015-10-01T18:54:39.805-0400 I SHARDING [conn1] distributed lock 'test' acquired, ts : 560db9afa3a189e276ebc8db
      [js_test:create_index_gle] 2015-10-01T18:54:39.901-0400 d20011| 2015-10-01T18:54:39.901-0400 D -        [conn13] User Assertion: 13435:not master and slaveOk=false
      [js_test:create_index_gle] 2015-10-01T18:54:39.902-0400 d20011| 2015-10-01T18:54:39.901-0400 D COMMAND  [conn13] assertion while executing command 'listDatabases' on database 'admin' with arguments '{ listDatabases: 1 }' and metadata '{ configsvr: { opTime: { ts: Timestamp 1443740079000|8, t: -1 } } }': 13435 not master and slaveOk=false
      [js_test:create_index_gle] 2015-10-01T18:54:39.902-0400 d20011| 2015-10-01T18:54:39.901-0400 I COMMAND  [conn13] command admin.$cmd command: listDatabases { listDatabases: 1 } ntoreturn:1 ntoskip:0 keyUpdates:0 writeConflicts:0 exception: not master and slaveOk=false code:13435 numYields:0 reslen:125 locks:{} protocol:op_command 0ms
      [js_test:create_index_gle] 2015-10-01T18:54:39.904-0400 s20015| 2015-10-01T18:54:39.903-0400 I SHARDING [conn1] distributed lock with ts: 560db9afa3a189e276ebc8db' unlocked.
      [js_test:create_index_gle] 2015-10-01T18:54:39.904-0400 s20015| 2015-10-01T18:54:39.903-0400 W SHARDING [conn1] could not refresh targeter :: caused by :: totalSize field not found in listDatabases
      [js_test:create_index_gle] 2015-10-01T18:54:39.905-0400 s20015| 2015-10-01T18:54:39.904-0400 W SHARDING [conn1] failed to get database config for test.system.indexes while checking for auto-split: NamespaceNotFound database test not found
      [js_test:create_index_gle] 2015-10-01T18:54:39.906-0400 assert: write or write concern failed: {
      [js_test:create_index_gle] 2015-10-01T18:54:39.906-0400 	"err" : "could not target full range of test.user; metadata not found",
      [js_test:create_index_gle] 2015-10-01T18:54:39.906-0400 	"code" : 26,
      [js_test:create_index_gle] 2015-10-01T18:54:39.906-0400 	"n" : 0,
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 	"shards" : [ ],
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 	"shardRawGLE" : {
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 	},
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 	"ok" : 1
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 }
      [js_test:create_index_gle] 2015-10-01T18:54:39.907-0400 _getErrorWithCode@src/mongo/shell/utils.js:23:13
      [js_test:create_index_gle] 2015-10-01T18:54:39.908-0400 doassert@src/mongo/shell/assert.js:13:14
      [js_test:create_index_gle] 2015-10-01T18:54:39.908-0400 assert.gleOK@src/mongo/shell/assert.js:462:9
      [js_test:create_index_gle] 2015-10-01T18:54:39.908-0400 @jstests/gle/create_index_gle.js:32:1
      [js_test:create_index_gle] 2015-10-01T18:54:39.908-0400 
      [js_test:create_index_gle] 2015-10-01T18:54:39.908-0400 2015-10-01T18:54:39.906-0400 E QUERY    [thread1] Error: write or write concern failed: {
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"err" : "could not target full range of test.user; metadata not found",
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"code" : 26,
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"n" : 0,
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"shards" : [ ],
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"shardRawGLE" : {
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	},
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 	"ok" : 1
      [js_test:create_index_gle] 2015-10-01T18:54:39.909-0400 } :
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 _getErrorWithCode@src/mongo/shell/utils.js:23:13
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 doassert@src/mongo/shell/assert.js:13:14
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 assert.gleOK@src/mongo/shell/assert.js:462:9
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 @jstests/gle/create_index_gle.js:32:1
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 
      [js_test:create_index_gle] 2015-10-01T18:54:39.910-0400 failed to load: jstests/gle/create_index_gle.js
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              schwerin Andy Schwerin
              Reporter:
              rassi J Rassi
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: