Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-15552

Errors writing to temporary collections during mapReduce command execution should be operation-fatal

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.6.6, 2.8.0-rc2
    • Component/s: MapReduce
    • Labels:
      None
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      Running the below Map Reduce job put the Mongodb instance in this state every time and its reproducible.

      mapFunc = function(){var k = [this.index[2],this.index[0],this.index[1]]; var v = {'Count':1, 'TotalWeight':this.value['Volume']}; emit(k,v);}
      reduceFunc = function(key,emits){total = {'Count':0, 'TotalWeight':0.0}; for (var i in emits){ total['Count'] += 1; total['TotalWeight'] += emits[i]['TotalWeight'];} return total;}
      db.RawData.mapReduce(mapFunc,reduceFunc,'Weights')

      Show
      Running the below Map Reduce job put the Mongodb instance in this state every time and its reproducible. mapFunc = function(){var k = [this.index[2],this.index[0],this.index[1]]; var v = {'Count':1, 'TotalWeight':this.value['Volume']}; emit(k,v);} reduceFunc = function(key,emits){total = {'Count':0, 'TotalWeight':0.0}; for (var i in emits){ total['Count'] += 1; total['TotalWeight'] += emits[i]['TotalWeight'];} return total;} db.RawData.mapReduce(mapFunc,reduceFunc,'Weights')

      Description

      A error in map-reduce job crashes the secondary servers, and prevents the secondaries from starting again. I know what the error is in my map function that causes the job to fail, but that shouldn't be leaving my mongodb instance in a irrecoverable state. The primary is up and running, but pushed to a secondary since that's the only replica that's running.

      The map function uses a list for a key, which is not supported. The unique index constraint is enforced on the last index of the list, which is not unique. Once I change it to a dictionary or concatenated string, it works just fine.

      Every time I try starting the secondary server, I get the same error "duplicate key error index" and it crashes. I had to wipe out the secondaries and let Mongodb do a clean sync, which came with a big downtime.

      This looks to be a mongodb bug. I am running a 3 replica set environment with 4 shards. All 4 shard servers in the 2 secondaries crashed with the same error.

      Any help is greatly appreciated. If there is a way to recover from current state, Please let me know as well.

      thanks!

      2014-10-07T00:28:32.159+0000 [conn18913] end connection 172.31.15.135:55897 (9 connections now open)
      2014-10-07T00:28:32.159+0000 [initandlisten] connection accepted from 172.31.15.135:55905 #18915 (10 connections now open)
      2014-10-07T00:28:32.160+0000 [conn18915]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
      2014-10-07T00:28:40.150+0000 [repl writer worker 1] ERROR: writer worker caught exception:  :: caused by :: 11000 insertDocument :: caused by :: 11000 E11000 duplicate key error index: ModelDatabase.tmp.mr.RawData_0.$_id_  dup key: { : "009020" } on: { ts: Timestamp 1412641720000|2, h: -267785287631189678, v: 2, op: "i", ns: "ModelDatabase.tmp.mr.RawData_0", o: { _id: [ "20111028", "0088", "009020" ], value: { Count: 6.0, TotalWeight: 7.0 } } }
      2014-10-07T00:28:40.150+0000 [repl writer worker 1] Fatal Assertion 16360
      2014-10-07T00:28:40.150+0000 [repl writer worker 1] 

        Attachments

        1. dump.tgz
          222 kB
        2. logs.tgz
          87 kB

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: