[SERVER-27312] Mapreduce does not run on Secondary in sharded clusters even with out: inline Created: 07/Dec/16 Updated: 07/Dec/16 Resolved: 07/Dec/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Dharshan Rangegowda | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
0. Nature of the job: public static final String reducefunction = "function(key, values) { return Array.sum(values); }"; (Code is attached) When executed on a Replica Set, following log can be seen on the Slave: 2016-11-23T15:05:26.735+0000 I COMMAND [conn671] command test.txns command: mapReduce { mapreduce: "txns", map: function() { emit(this.custid, this.txnval); } , reduce: function(key, values) { return Array.sum(values); }, out: { inline: 1 }, query: null, sort: null, finalize: null, scope: null, verbose: true } planSummary: COUNT keyUpdates:0 writeConflicts:0 numYields:7 reslen:4331 locks:{ Global: { acquireCount: { r: 44 } }, Database: { acquireCount: { r: 3, R: 19 } }, Collection: { acquireCount: { r: 3 } } } protocol:op_query 124msThe out : {inline : 1} ensure that it hits the slave. When executed on a shard: mongos> db.txns.getShardDistribution() Shard Shard-0 at Shard-0/SG-shard3-281.devservers.mongodirector.com:27017,SG-shard3-282.devservers.mongodirector.com:27017 data : 498KiB docs : 9474 chunks : 3 estimated data per chunk : 166KiB estimated docs per chunk : 3158 Shard Shard-1 at Shard-1/SG-shard3-284.devservers.mongodirector.com:27017,SG-shard3-285.devservers.mongodirector.com:27017 data : 80KiB docs : 1526 chunks : 3 estimated data per chunk : 26KiB estimated docs per chunk : 508 Totals data : 579KiB docs : 11000 chunks : 6 Shard Shard-0 contains 86.12% data, 86.12% docs in cluster, avg obj size on shard : 53B Shard Shard-1 contains 13.87% data, 13.87% docs in cluster, avg obj size on shard : 53B Shard 0 Mongos doesn't log the MapReduce job at all Slave only logs the deletion of the tmp collection. Primary: 2016-11-24T08:46:30.828+0000 I COMMAND [conn357] command test.$cmd command: mapreduce.shardedfinish { mapreduce.shardedfinish: { mapreduce: "txns", map: function() { emit(this.custid, this.txnval); }, reduce: function(key, values) { return Array.sum(values); } , out: { in line: 1 }, query: null, sort: null, finalize: null, scope: null, verbose: true, $queryOptions: { $readPreference: { mode: "secondary" } } }, inputDB: "test", shardedOutputCollection: "tmp.mrs.txns_1479977190_0", shards: { Shard-0/SG-shard3-281.devservers.mongodirector.com , counts: { input: 9474, emit: 9474, reduce: 909, output: 101 }, ok: 1.0, $gleS }, Shard-1/SG-shard3-284.devservers.mongodirector.com:27017,SG-shard3-285.devservers.mongodirector.com:27017: { result: "tmp.mrs.txns_1479977190_0", timeMillis: 71, timing: , counts: { input: 1526, emit: 1526, reduce: 197, output: 101 }, ok: 1.0, $gleStats: { lastOpTime: Timestamp 1479977190000|103, electionId: ObjectId('7fffffff0000000000000001') } } }, shardCounts: { Sha , Shard-1/SG-shard3-284.devservers.mongodirector.com:27017,SG-shard3-285.devservers.mongodirector.com:27017: { inpu t: 1526, emit: 1526, reduce: 197, output: 101 }}, counts: { emit: 11000, input: 11000, output: 202, reduce: 1106 }} keyUpdates:0 writeConflicts:0 numYields:0 reslen:4368 locks:{ Global: { acquireCount: { r: 2 }}, Database: { acquireCount: { r: 1 } }, Collection: { acqu } } protocol:op_command 115ms Looks like temp collections are created when it is a sharded database/collection. Couldn't find anything specific about the behavior in the documentation (https://docs.mongodb.com/v3.2/core/map-reduce-sharded-collections/). Shard 1, Primary: Secondary: |
| Comments |
| Comment by Stennie Steneker (Inactive) [ 07/Dec/16 ] |
|
Hi Dharshan, This is currently expected behaviour for Map/Reduce via mongos; please watch/upvote Thanks, |