Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35214

Invariant failure starting up mongos after automated restore from backup

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.0-rc0
    • Component/s: Sharding
    • Labels:
      None
    • ALL
    • Hide

      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.

      Show
      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.
    • Sharding 2018-06-18

      When doing a sharded cluster automated backup restore on 4.0.0-rc0, when we try to restart the mongos after the rest of the cluster has been restored, we get the following invariant failure in the mongos logs:

      2018-05-24T19:36:22.400+0000 I SHARDING [LogicalSessionCacheRefresh] Refreshing chunks for collection config.system.sessions based on version 0|0||000000000000000000000000 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] Invariant failure _shardId == _history.front().getShard() src/mongo/s/chunk.cpp 67 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] ***aborting after invariant() failure
      

      Full logs attached.

      This happens for about 20 minutes (we keep retrying the start every 3 minutes or so), and then it successfully starts up.

      Can try to narrow down a better repro if needed – this is from our E2E test runs, so there's not as much information as usual. If this is not sufficient information, I'll generate a repro tomorrow that we can look at more closely. kaloian.manassiev suggested I file a ticket for the failure.

        1. config_40WT (1).tar.gz
          34 kB
        2. mongos_after_restore_fassert.log
          44 kB

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            louisa.berger@mongodb.com Louisa Berger
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: