[SERVER-2531] map/reduce output options need to work on sharded output collections out=(merge|reduce) Created: 10/Feb/11 Updated: 12/Jul/16 Resolved: 27/Jun/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | MapReduce |
| Affects Version/s: | 1.7.5 |
| Fix Version/s: | 1.9.1 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Alvin Richards (Inactive) | Assignee: | Antoine Girbal |
| Resolution: | Done | Votes: | 6 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
Right now you can't merge or reduce into a sharded collection. |
| Comments |
| Comment by auto [ 29/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by auto [ 27/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by auto [ 27/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by Antoine Girbal [ 27/Jun/11 ] |
|
ganesan, Process is as follows, and involves only 1 mongos:
One thing I still have to figure out is how we want to handle splitting / migrating of the chunks. I'll try to put a diagram together soon |
| Comment by auto [ 27/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by ganesan pandurangan [ 27/Jun/11 ] |
|
Hi Antoine, Is this ticket complete, Can I use this code for testing ? Also, on the mongos part - what is the expected behavior if there are multiple mongos ? Will the work be shared across the multiple mongos, or will the work be done by the mongos that initially posted the map reduce command ?, Does it put any pressure on the mongos ( like memory or cpu requriements) Can you please send me a small diagram of how things work with all your changes? regards |
| Comment by Antoine Girbal [ 27/Jun/11 ] |
|
added REDUCE mode for sharded output.
|
| Comment by auto [ 27/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by auto [ 22/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by Antoine Girbal [ 13/Jun/11 ] |
|
It was quite a bit of work to rip out some mr code and make it executed in mongos. Todo:
INLINE mode will not be supported cause does not make sense. |
| Comment by auto [ 13/Jun/11 ] |
|
Author: {u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}Message: |
| Comment by Antoine Girbal [ 26/May/11 ] |
|
ok sounds like there are several issues here:
The 1st goal of this ticket is to make it possible to have sharded output.
|
| Comment by Jalmari Raippalinna [ 03/May/11 ] |
|
Workaround: after first mapReduce seems to fix this. It seems that out collections do not have _id index automatically created. |
| Comment by Jalmari Raippalinna [ 03/May/11 ] |
|
This happens with 1.8.1 when output collection is not sharded, but is used in sharded environment. Our environment is MongoDB 1.8.1 with 2 shards and 3 replica sets each (6 servers). Map/Reduces are done against sharded input collections and output into temporary non-existing collection with out: { reduce: mr_temp_collection }Will get this: Tue May 3 10:18:08 uncaught exception: map reduce failed:{ |
| Comment by Andy Gregorowicz [ 12/Apr/11 ] |
|
There also seems to be an issue when output collections are not sharded, but the M/R job is run more than once. When output is set to reduce, the first time the M/R job is run, it will succeed, but remove all indexes that were set on the collection and leave one called 0_1. Running the M/R job a second time will cause a "Not an index cursor" error. When the output is set to merge, the first job succeeds and on the second run of the M/R job one of the shard servers will consume 100% of a CPU core and never return. I am seeing this behavior on 1.8.1. |