[SERVER-3627] sharded map-reduce output should be parallelized and properly distribute chunks Created: 17/Aug/11  Updated: 11/Jul/16  Resolved: 21/Dec/11

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: 2.1.0

Type: Bug Priority: Major - P3
Reporter: Chris Westin Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tested Vista


Issue Links:
Depends
is depended on by SERVER-3253 aggregation: unsharded support $out Closed
is depended on by SERVER-10097 aggregation: support $out on a sharde... Closed
Related
is related to SERVER-2531 map/reduce output options need to wor... Closed
Operating System: ALL
Participants:

 Description   

See QA-12 for the test case.



 Comments   
Comment by Antoine Girbal [ 21/Dec/11 ]

verified:

  • proper output
  • initial splitting of chunks
  • balancing of chunks
  • further splitting on incremental MR
  • exclusive locking of the output collection

tests are in bigMapReduce.js and mrShardedOutput.js.

Comment by auto [ 21/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: lookup of chunk was incorrect when checking on split
Branch: master
https://github.com/mongodb/mongo/commit/7525b291d104c7acd1823905e8cb269da49bd767

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: disable migration during mr sharded output. More complete testing.
Branch: master
https://github.com/mongodb/mongo/commit/1d27cf53df4ac6e3fa1e3f8ac09da5d42ac9421a

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: shard collection and initial chunks much simpler. Test now verifies proper balancing. Test now passes reliably.
Branch: master
https://github.com/mongodb/mongo/commit/7e4a1549711a3d28cc471e106047e01fb9e9728a

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: disable migration during mr sharded output. More complete testing.
Branch: master
https://github.com/mongodb/mongo/commit/1d27cf53df4ac6e3fa1e3f8ac09da5d42ac9421a

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: shard collection and initial chunks much simpler. Test now verifies proper balancing. Test now passes reliably.
Branch: master
https://github.com/mongodb/mongo/commit/7e4a1549711a3d28cc471e106047e01fb9e9728a

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: remove passing of shard results back to each shard, can be very big object
Branch: master
https://github.com/mongodb/mongo/commit/79a35c40068c27d9f5324773aac288977762a9ca

Comment by auto [ 17/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: ability to assign shards to chunks
Branch: master
https://github.com/mongodb/mongo/commit/ff24938b5aa495e4bdee971085151534852e627c

Comment by auto [ 15/Dec/11 ]

Author:

{u'login': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: buildbot bigMapReduce.js can fail until SERVER-3627 resolved, noted
Branch: master
https://github.com/mongodb/mongo/commit/955d3591ad35ec9ba2e26f624a622cc826c5360b

Comment by auto [ 08/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: do not use at() on map
Branch: master
https://github.com/mongodb/mongo/commit/afa5b9cc3d4c9325e59269f4d1d34b32283ebeba

Comment by auto [ 07/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: chunk range info no longer relayed by mongos, mongod figures it out
Branch: master
https://github.com/mongodb/mongo/commit/6457aaf593b0ba8c765e261bad8deef2891ab74a

Comment by auto [ 01/Dec/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: mongod reports chunk sizes, mongos splits if needed.
Branch: master
https://github.com/mongodb/mongo/commit/66a2b9c2647726cc603868e981681c94d22e4d3a

Comment by auto [ 29/Nov/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: assert failed in inline mode
Branch: master
https://github.com/mongodb/mongo/commit/f853f0d051106dcbcf480f298d9b45500282bb9a

Comment by auto [ 24/Nov/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: refactor post processing of parallel MR jobs to be parallelize and allow for sharded output
Branch: master
https://github.com/mongodb/mongo/commit/525634baa56f6de7a45a1fe352f06b1309d08728

Comment by auto [ 18/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3627: splitting works correctly now. MR remembers which chunks are touched and runs split after post-process
Branch: master
https://github.com/mongodb/mongo/commit/125103fc008b48be497f0418151b852e273e9b65

Comment by Antoine Girbal [ 13/Oct/11 ]

Basically the problematic code is after all M/R have run on each shard.
Mongos needs to gather docs in order and insert them to the right shards based on the output collection.
This is made more complex by the MERGE and REDUCE modes where there is an existing final collection.
2 solutions:

1. Send the new records to a temp collection that is aligned with the final collection's sharding
Pros:

  • uses existing code for post-processing
  • maintain all mode functionalities, including the REPLACE atomicity.
  • in REDUCE mode, post process reducing is done on the shard

Cons:

  • not easy to align sharding. The idea is to use the final collection's NS for chunk lookup. This works in normal case but can be broken by chunk migration, and implementing splitting is difficult

2. Simplify the output modes with no atomicity guaranteed.
With this solution, mongos inserts / updates records directly into the final collection.
For replace mode, drop the final collection and recreate it and insert records.
For merge mode, just upsert records.
For reduce mode, pull the existing record, reduce and update.

Pros:

  • no sharding complexity, works well with splitting / migrating
  • replace / merge mode are faster

Cons:

  • reduce mode is much slower.
  • replace mode is non atomic and collection does not exist during processing

So far solution #1 was implemented to maintain most functionality.
But the sharding complexities may call for #2 instead.

Comment by Antoine Girbal [ 13/Oct/11 ]

Ticket was reopened because it would not split properly on certain servers.
I think I know where the problem is.
The objects are inserted into a temp collection since post processing needs to be done within each mongod (same as with non sharded MR output).
But the temp collection's sharding needs to match exactly the final collection's sharding.
So it uses the final collection NS to lookup the chunks and send the insert to the right shard, then does post processing.
This doesnt work well with splitting/migrating, although dont understand why it is splitting on some boxes then.
Maybe it's an artifact of the fact that test runs MR several times.

Comment by auto [ 11/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3627: disable test, not passing on buildbot
Branch: master
https://github.com/mongodb/mongo/commit/f14315265113e7e9a00b06125b6b285dd71ab162

Comment by Antoine Girbal [ 10/Oct/11 ]

added test

Comment by auto [ 10/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3627: sharded map-reduce output isn't being split into chunks
Branch: master
https://github.com/mongodb/mongo/commit/615e95459ed462e8fe20eaaa5d888c611427ada8

Comment by Eliot Horowitz (Inactive) [ 22/Aug/11 ]

Should be easy to pre-split.

Comment by Antoine Girbal [ 22/Aug/11 ]

yes when I put together the methods to do internal insert/update for sharded system, I had to remove a couple things because the request and dbmessage objects do not exist.
The lines are:
r.gotInsert();
if ( r.getClientInfo()->autoSplitOk() )
c->splitIfShould( o.objsize() );

For now the workaround is obviously to presplit, but not ideal.
Will figure out a way to put them in.
AG

Generated at Thu Feb 08 03:03:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.