[SERVER-14269] Map Reduce fails with duplicate-key when output is 'merge' and sharded. Created: 16/Jun/14  Updated: 25/Jun/15  Resolved: 27/Aug/14

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: 2.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Santiago Alessandri Assignee: Randolph Tan
Resolution: Duplicate Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File test.js    
Issue Links:
Duplicate
duplicates SERVER-7926 Map Reduce with sharded output can ap... Closed
Related
is related to SERVER-15024 mapReduce output to sharded collectio... Closed
Operating System: ALL
Steps To Reproduce:

Run mongo < test.js on a sharded cluster twice.

Original steps
  • Create a sharded input collection.
  • Execute a map reduce with sharded output and be sure that the output collection has more than one chunk.
  • Repeat the execution with more data in the input collection in order to make the output grow but also to have results with the same key.
Participants:

 Description   

There is an issue when using map reduce with sharded output using merge mode.

If there is more than one chunk in the output collection and some of the map reduce values have a key already stored in the result collection, the map reduce fails stating:
"exception: insertDocument :: caused by :: 11000 E11000 duplicate key error index"

At first I thought it might be because I was using the same collection as input and as output. But it also happens when using different collections.

This doesn't happen if the output collection is unsharded or if it only has one chunk.

The map reduce was executed through mongo and also through pymongo with the same behavior.

This bug might not happen the first time you execute a map reduce on the collection with already stored keys. But after several executions that make the output collection grow and get divided into more chunks the bug shows up.

I haven't tried what happens when the input collection is not sharded.



 Comments   
Comment by Randolph Tan [ 25/Aug/14 ]

The duplicate error happens at the last phase of the sharded map reduce where the shards upserts all the documents that it thinks that it owns to the final output collection. Because of SERVER-7926, the shards can potentially be using the outdated chunk boundaries and actually try to upsert documents it doesn't own to it's own shard. If the document already exists, it will fail the upsert with the duplicate key error. This is because the internal runner (in the test case, the idhack runner was used) will filter out the document when it executes the "query" part of the upsert since the shard technically does not own it and will proceed to insert it. This will then trigger the duplicate key error by the time it tries to update the index entries since it already exists.

Comment by Ramon Fernandez Marina [ 25/Jul/14 ]

Alife, salessandri, thanks both for your prompt responses. I'm able to reproduce with the latest stable version so we're good to go.

Comment by Serge A Terekhov [X] [ 25/Jul/14 ]

Dear Ramon, unfortunately the environment is related to data owned by someone else, so I can't post the original case. I'll restore June version of our code and 'll repeat tests with 2.6.3, and inform you about results (it takes time though). What I can post is server log (large, better to be sent via email), and also I can send you (personally) our map and reduce js scripts. Please, drop me a line to alife at narod.ru Thanks!

Comment by Ramon Fernandez Marina [ 25/Jul/14 ]

Nevermind, I'm able to reproduce in 2.6.2 and 2.6.3; the first run of the reproducer above doesn't show the problem, but the second one does.

Comment by Ramon Fernandez Marina [ 25/Jul/14 ]

Alife, do you have a reproducer for 2.6.2 you can upload?

Comment by Santiago Alessandri [ 25/Jul/14 ]

Thanks, Ramon.

I will re-enable the script I was using that generated the bug on the new mongo version. If it still breaks I will let you know and I will also try to generate a script like this and post it.

Also, if it doesn't break I will post it here.

Comment by Ramon Fernandez Marina [ 25/Jul/14 ]

salessandri, I'm able to reproduce the issue on 2.6.1 using the following script:

db.getSiblingDB("config").settings.update({_id:"chunksize"},{$set:{"value":1}})
db.foo.drop()
sh.enableSharding("test")
sh.shardCollection("test.foo", {_id:1})
 
for (i=0; i<30000; i++) {
    db.foo.insert({x:Math.floor(1000*Math.random())})
}
 
db.out.drop()
var map1 = function() { emit(this._id, this.x); };
var red1 = function(id, x) { x };
db.foo.mapReduce( map1, red1, { out: {merge: "out", sharded: true}})
db.foo.mapReduce( map1, red1, { out: {merge: "out", sharded: true}})

However, I can't reproduce the problem on 2.6.2 or 2.6.3. Can you please try upgrading to 2.6.3 and see if you still have the problem? If you do, having a reproducer would be very helpful diagnosing this issue.

Comment by Serge A Terekhov [X] [ 18/Jun/14 ]

Error is still present in 2.6.2 version, output mode

{reduce: coll, sharded: 1}
Comment by Serge A Terekhov [X] [ 16/Jun/14 ]

Same happens also in my case with {out: {reduce:<collname>, sharded: 1}}

Message:

MR post processing failed: { result: { db: "repu", collection: "stat" }, chunkSizes: [ { _id: MinKey }, 5759761 ], errmsg: "exception: insertDocument :: caused by :: 11000 E11000 duplicate key error index: repu.stat.$_id_  dup key: { : "B1000028748293459927" }", code: 11000, ok: 0.0 }

Multiple warnings from shard log preceeding the error:

2014-06-13T13:34:37.657+0400 [conn2] warning: ClientCursor::staticYield can't unlock b/c of recursive lock ns:  top: { opid: 141474, active: true, secs_running: 1, op: "query", ns: "repu.stat", query: { $msg: "query not recording (too large)" }, client: "92.36.25.144:57829", desc: "conn2", connectionId: 2, locks: { ^: "W" }, waitingForLock: false, msg: "m/r: reduce post processing M/R Reduce Post Processing Progress: 14760/24703 59%", progress: { done: 14760, total: 24703 }, numYields: 0, lockStats: { timeLockedMicros: { r: 0, w: 218341 }, timeAcquiringMicros: { r: 0, w: 39394 } } }

Note: same job running in non-sharded environments (single Mongod) is perfectly fine.

Generated at Thu Feb 08 03:34:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.