[SERVER-5357] Map/Reduce operation writes output data in primary node of replica set but not in secondary nodes Created: 22/Mar/12  Updated: 11/Jul/16  Resolved: 06/Apr/12

Status: Closed
Project: Core Server
Component/s: MapReduce, Replication
Affects Version/s: 2.0.3
Fix Version/s: 2.0.5, 2.1.1

Type: Bug Priority: Major - P3
Reporter: Corentin Guy Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: mapreduce, replicaset
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

GNU Debian


Issue Links:
Depends
Operating System: Linux
Participants:

 Description   

In a sharded / replica set environment, map/reduce output data are not written in secondary nodes :

I have 9 nodes (6 nodes + 3 arbiters) shared between 3 replica sets (rs1, rs2, rs3). My map/reduce operation process a sharded collection, and write output result in a normal collection (non-sharded).

As result, I have my output collection created in primay and secondary node, but only primary node contains data.

I didn't found any information in logs..

I can show you replication status :

PRIMARY> db.count_dm_4f6aef18d4ee7771b1a591e6.find(); // Returns my element

{ "_id" : "4f6aef18d4ee7771b1a591e6", "value" : 453 }

PRIMARY> db.printReplicationInfo()
configured oplog size: 6966.868750000001MB
log length start to end: 85347secs (23.71hrs)
oplog first event time: Wed Mar 21 2012 11:21:07 GMT+0100 (CET)
oplog last event time: Thu Mar 22 2012 11:03:34 GMT+0100 (CET)
now: Thu Mar 22 2012 11:03:35 GMT+0100 (CET)

SECONDARY> db.count_dm_4f6aef18d4ee7771b1a591e6.find(); // Returns nothing ...
SECONDARY> db.printReplicationInfo()
configured oplog size: 6792.466796875MB
log length start to end: 85358secs (23.71hrs)
oplog first event time: Wed Mar 21 2012 11:21:07 GMT+0100 (CET)
oplog last event time: Thu Mar 22 2012 11:03:45 GMT+0100 (CET)
now: Thu Mar 22 2012 11:03:46 GMT+0100 (CET)



 Comments   
Comment by auto [ 24/Apr/12 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: backport of SERVER-5357
Branch: v2.0
https://github.com/mongodb/mongo/commit/d861e27e783406217a5d7f945a39169e5ae250a2

Comment by auto [ 06/Apr/12 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-5357: cleanup following review
Branch: master
https://github.com/mongodb/mongo/commit/7ded96c40088c935aa4bdc1381a1fab1078033b3

Comment by auto [ 06/Apr/12 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-5357: Map/Reduce operation writes output data in primary node of replica set but not in secondary nodes
Branch: master
https://github.com/mongodb/mongo/commit/a139832baaf9a4e1e3a4658ab1ced12f0e68b41c

Comment by Corentin Guy [ 04/Apr/12 ]

Ok, tell me when the fix will be released (in 2.1 and 2.0.x branches). Thank you !

Comment by Antoine Girbal [ 04/Apr/12 ]

@Corentin the bug is identify and fix is easy.
It will be in 2.1 branch shortly, then we need to consider backporting.

Comment by Corentin Guy [ 03/Apr/12 ]

There the full line for MR : db.Hit.mapReduce(m, r,

{"out":"Tagdm"}

); with
m = function m() {if (this.app == "dm" && this.params['tag'] != null) {emit(this.params['tag'], null);}}
and r = function( key , values )

{ return; }

The previous example is quite different than the one exposed in the ticket description, but it is simpler.

There the sharding status :
mongos> db.printShardingStatus()
– Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "rs1", "host" : "rs1/192.168.0.5:10001,192.168.0.6:10001" } { "_id" : "rs2", "host" : "rs2/192.168.0.7:10001,192.168.0.8:10001" } { "_id" : "rs3", "host" : "rs3/192.168.0.10:10001,192.168.0.9:10001" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "integration_stats", "partitioned" : true, "primary" : "rs3" }

integration_stats.Hit chunks:
rs2 18
rs3 18
rs1 19
too many chunks to print, use verbose if you want to force print

{ "_id" : "*", "partitioned" : false, "primary" : "rs3" }

And finally stats of sharded source collection :

mongos> db.Hit.stats();
{
"sharded" : true,
"flags" : 1,
"ns" : "integration_stats.Hit",
"count" : 4507426,
"numExtents" : 64,
"size" : 3051345224,
"storageSize" : 3447533568,
"totalIndexSize" : 155066016,
"indexSizes" :

{ "_id_" : 155066016 }

,
"avgObjSize" : 676.9595826975307,
"nindexes" : 1,
"nchunks" : 55,
"shards" : {
"rs1" : {
"ns" : "integration_stats.Hit",
"count" : 1854203,
"size" : 1259226608,
"avgObjSize" : 679.1201438030248,
"storageSize" : 1457278976,
"numExtents" : 23,
"nindexes" : 1,
"lastExtentSize" : 252166144,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 61262768,
"indexSizes" :

{ "_id_" : 61262768 }

,
"ok" : 1
},
"rs2" : {
"ns" : "integration_stats.Hit",
"count" : 1282959,
"size" : 860748452,
"avgObjSize" : 670.9087757286086,
"storageSize" : 986931200,
"numExtents" : 20,
"nindexes" : 1,
"lastExtentSize" : 175112192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 45221456,
"indexSizes" :

{ "_id_" : 45221456 }

,
"ok" : 1
},
"rs3" : {
"ns" : "integration_stats.Hit",
"count" : 1370264,
"size" : 931370164,
"avgObjSize" : 679.7012575678847,
"storageSize" : 1003323392,
"numExtents" : 21,
"nindexes" : 1,
"lastExtentSize" : 174579712,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 48581792,
"indexSizes" :

{ "_id_" : 48581792 }

,
"ok" : 1
}
},
"ok" : 1
}

I understand in your last answer that bug seems to really exist. I don't really understand the origin of the bug, but it doesn't matter. So, how can I help for the correction ? Is the bug trully identified ?

Comment by Antoine Girbal [ 02/Apr/12 ]

it looks like there is a bug there indeed, which I think as always been in mongo.
Basically during the final reduce on the target shard, the inc and temp collections are one and the same since entries wont need further reducing.
The code uses _insertToInc to add the final results, and that bypasses the oplog.
Last, the inc/temp collection is renamed to the final name, and wont have data on secondaries.

Comment by Antoine Girbal [ 02/Apr/12 ]

can you provide:

  • the full line you call for MR
  • output of db.printShardingStatus()
  • stats of the sharded source collection
Comment by Corentin Guy [ 02/Apr/12 ]

I've upgraded MongoDB version from 2.0.3 to 2.0.4, the "bug" is still there.

Comment by Corentin Guy [ 27/Mar/12 ]

Please see following discussion on Google Group (some details in answers) : https://groups.google.com/forum/?fromgroups&pli=1#!topic/mongodb-user/NmnHOxT04A4

Generated at Thu Feb 08 03:08:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.