Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35214

Invariant failure starting up mongos after automated restore from backup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 4.0.0-rc0
    • Fix Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.

      Show
      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.
    • Sprint:
      Sharding 2018-06-18

      Description

      When doing a sharded cluster automated backup restore on 4.0.0-rc0, when we try to restart the mongos after the rest of the cluster has been restored, we get the following invariant failure in the mongos logs:

      2018-05-24T19:36:22.400+0000 I SHARDING [LogicalSessionCacheRefresh] Refreshing chunks for collection config.system.sessions based on version 0|0||000000000000000000000000 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] Invariant failure _shardId == _history.front().getShard() src/mongo/s/chunk.cpp 67 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] ***aborting after invariant() failure
      

      Full logs attached.

      This happens for about 20 minutes (we keep retrying the start every 3 minutes or so), and then it successfully starts up.

      Can try to narrow down a better repro if needed – this is from our E2E test runs, so there's not as much information as usual. If this is not sufficient information, I'll generate a repro tomorrow that we can look at more closely. Kaloian Manassiev suggested I file a ticket for the failure.

        Attachments

        1. config_40WT (1).tar.gz
          34 kB
        2. mongos_after_restore_fassert.log
          44 kB

          Activity

            People

            Assignee:
            kaloian.manassiev Kaloian Manassiev
            Reporter:
            louisa.berger Louisa Berger
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: