Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20725

Mongo 3.0.6 secondary crashes when while replicating map-reduce with a background index

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.6
    • Labels:
      None
    • ALL
    • Hide

      1. Setup replica set with at least 2-3 servers (adjust voting to be odd) where the secondary is at least 100-300 ms away from primary.
      2. Create 'source' collection with some data (try 100 records)
      3. Create 'output' collection with background index on one field (in our case composite index).
      4. Perform map-reduce with 'reduce' output to the output collection every 10 seconds (in our case 15 minute).

      Show
      1. Setup replica set with at least 2-3 servers (adjust voting to be odd) where the secondary is at least 100-300 ms away from primary. 2. Create 'source' collection with some data (try 100 records) 3. Create 'output' collection with background index on one field (in our case composite index). 4. Perform map-reduce with 'reduce' output to the output collection every 10 seconds (in our case 15 minute).

      There seems to be a bug in mongo when using background indexes on collections that are output of map-reduce. In some cases secondary mongos crash.

      Details:

      Our setup is 1 primary with 3 secondaries. Each server is in different geographical region (NA, Asia, Europe) so there's some network lag.

      A map-reduce with 'reduce' to a collection is performed periodically. Occasionally some secondaries crash while building a background index on a temporary collection (created by server for map-reduce) when the collection gets dropped before the index is built.

      Map reduce creates multiple temporary collections with same indexes as the main collection where it stores intermediary data while computing and reducing/merging results.

      These temp collections (unnecessary) get replicated on all secondaries. The indexes are also created in the background as the original index. In some cases (depending on network lag) the temp collection is processed and deleted on the primary before the secondary has finished creating the index and as a result it crashes.

      Or, because of network lag, the create collection, create index in background and drop collection commands come at the same time and the secondary crashes on building index after the collection was already dropped.

      Work-around:

      Making the map-reduce collection index in foreground forces the temp collection indexes to be built in foreground and thus block any operations on the collection while the index is created. This hopefully would block the drop command till the index is created. These collections are always empty when the index is being created so there's no performance impact.

      So far we haven't had any crashes after changing the indexes to foreground.

      While this happens while performing map-reduce there might be cases where a script can perform similar create/drop collection very fast and reproduce the problem.

      It would be also nice if the temp collections created by map-reduce process are recognized as temp and not replicated to secondaries.

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            plamen Plamen Parvanov
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: