Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21583

ApplyOps background index creation may deadlock

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      Linux
    • Backport Completed:
    • Steps To Reproduce:
      Hide
      • new replicaset without any data
      • standalone mongod on port 30005 (just for demonstration)
      • on replicaset primary:

         
        db.foobar.createIndex({"a": 1 }, {background: true})
        db.foobar.insert({"a":NumberLong("1444173423471"),"_id":"1444173420000-3"})
        

      • mongodump of oplog

        mongodump -d local -c oplog.rs --port 30001 -o dumptest/
        

      • reformat dump:

        mv dumptest/local/oplog.rs.bson dumptest/oplog.bson; rm -rf dumptest/local
        

      • mongorestore to replay oplog:

        mongorestore -v  --port 30005 --oplogReplay dumptest/
        

      Show
      new replicaset without any data standalone mongod on port 30005 (just for demonstration) on replicaset primary: db.foobar.createIndex({"a": 1 }, {background: true}) db.foobar.insert({"a":NumberLong("1444173423471"),"_id":"1444173420000-3"}) mongodump of oplog mongodump -d local -c oplog.rs --port 30001 -o dumptest/ reformat dump: mv dumptest/local/oplog.rs.bson dumptest/oplog.bson; rm -rf dumptest/local mongorestore to replay oplog: mongorestore -v --port 30005 --oplogReplay dumptest/
    • Sprint:
      Repl E (01/08/16)

      Description

      I tried to replay the oplog on one server to recover however the mongorestore did kept running indefinitely, on the server many op are running trying to wait for a lock, and when trying to stop the server, it's not responsive, so I have to run kill-9 against it.

      1. mongod.log
        21 kB
        Ramon Fernandez
      2. oplog.bson
        0.4 kB
        Ramon Fernandez

        Issue Links

          Activity

          Hide
          benety.goh Benety Goh added a comment -

          Seems to be an existing validation issue in 3.2 and 3.0. You can trigger the invariant in 3.2.0 and 3.0.8 by running the same command without background: true:

          db.adminCommand({
              applyOps: [{
                      'op': 'i',
                      'ns': db.getName() + '.system.indexes',
                      'o': {
                          ns: 'u',
                          key: { b: 1 },
                          name: 'b_1'
                      }
                  }]
          });
          

          We should probably fix this.

          Show
          benety.goh Benety Goh added a comment - Seems to be an existing validation issue in 3.2 and 3.0. You can trigger the invariant in 3.2.0 and 3.0.8 by running the same command without background: true : db.adminCommand({ applyOps: [{ 'op': 'i', 'ns': db.getName() + '.system.indexes', 'o': { ns: 'u', key: { b: 1 }, name: 'b_1' } }] }); We should probably fix this.
          Hide
          kamran.khan Kamran K. (Inactive) added a comment -

          Good catch. I filed that issue separately as SERVER-22109.

          Show
          kamran.khan Kamran K. (Inactive) added a comment - Good catch. I filed that issue separately as SERVER-22109 .
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

          Message: SERVER-21583 fixed background index options in background index replsets test

          (cherry picked from commit 54dfb6df908be81c50587a854964551ad72110cc)
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/2f08ba3bcf0b4aabf6609ed5e6fff36cfb227605

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'} Message: SERVER-21583 fixed background index options in background index replsets test (cherry picked from commit 54dfb6df908be81c50587a854964551ad72110cc) Branch: v3.2 https://github.com/mongodb/mongo/commit/2f08ba3bcf0b4aabf6609ed5e6fff36cfb227605
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

          Message: SERVER-21583 applyOperation_inlock builds background indexes in foreground if temp release fails

          (cherry picked from commit edc5989d36f6dea86ffe034c414e97e98549004e)
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/229c741beedd812afe17e1be3e18958a88b40203

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'} Message: SERVER-21583 applyOperation_inlock builds background indexes in foreground if temp release fails (cherry picked from commit edc5989d36f6dea86ffe034c414e97e98549004e) Branch: v3.2 https://github.com/mongodb/mongo/commit/229c741beedd812afe17e1be3e18958a88b40203
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

          Message: SERVER-21583 added applyOps test case for creating foreground and background indexes

          (cherry picked from commit 4ed0efa3368a89cb808c62b2d9e916acf8202ff7)
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/89b08ef60181d725d285c5dfb32197cafd47e361

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'} Message: SERVER-21583 added applyOps test case for creating foreground and background indexes (cherry picked from commit 4ed0efa3368a89cb808c62b2d9e916acf8202ff7) Branch: v3.2 https://github.com/mongodb/mongo/commit/89b08ef60181d725d285c5dfb32197cafd47e361

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile