There seems to be a bug in mongo when using background indexes on collections that are output of map-reduce. In some cases secondary mongos crash.

Details:

Our setup is 1 primary with 3 secondaries. Each server is in different geographical region (NA, Asia, Europe) so there's some network lag.

A map-reduce with 'reduce' to a collection is performed periodically. Occasionally some secondaries crash while building a background index on a temporary collection (created by server for map-reduce) when the collection gets dropped before the index is built.

Map reduce creates multiple temporary collections with same indexes as the main collection where it stores intermediary data while computing and reducing/merging results.

These temp collections (unnecessary) get replicated on all secondaries. The indexes are also created in the background as the original index. In some cases (depending on network lag) the temp collection is processed and deleted on the primary before the secondary has finished creating the index and as a result it crashes.

Or, because of network lag, the create collection, create index in background and drop collection commands come at the same time and the secondary crashes on building index after the collection was already dropped.

Work-around:

Making the map-reduce collection index in foreground forces the temp collection indexes to be built in foreground and thus block any operations on the collection while the index is created. This hopefully would block the drop command till the index is created. These collections are always empty when the index is being created so there's no performance impact.

So far we haven't had any crashes after changing the indexes to foreground.

While this happens while performing map-reduce there might be cases where a script can perform similar create/drop collection very fast and reproduce the problem.

It would be also nice if the temp collections created by map-reduce process are recognized as temp and not replicated to secondaries.