Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35214

Invariant failure starting up mongos after automated restore from backup

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Works as Designed
    • Icon: Major - P3 Major - P3
    • None
    • 4.0.0-rc0
    • Sharding
    • None
    • ALL
    • Hide

      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.

      Show
      See the spec for sharded cluster automated restore for reference on exact procedure for sharded cluster restore from backup.
    • Sharding 2018-06-18

    Description

      When doing a sharded cluster automated backup restore on 4.0.0-rc0, when we try to restart the mongos after the rest of the cluster has been restored, we get the following invariant failure in the mongos logs:

      2018-05-24T19:36:22.400+0000 I SHARDING [LogicalSessionCacheRefresh] Refreshing chunks for collection config.system.sessions based on version 0|0||000000000000000000000000 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] Invariant failure _shardId == _history.front().getShard() src/mongo/s/chunk.cpp 67 
      2018-05-24T19:36:22.400+0000 F - [ConfigServerCatalogCacheLoader-0] ***aborting after invariant() failure
      

      Full logs attached.

      This happens for about 20 minutes (we keep retrying the start every 3 minutes or so), and then it successfully starts up.

      Can try to narrow down a better repro if needed – this is from our E2E test runs, so there's not as much information as usual. If this is not sufficient information, I'll generate a repro tomorrow that we can look at more closely. kaloian.manassiev suggested I file a ticket for the failure.

      Attachments

        1. config_40WT (1).tar.gz
          34 kB
        2. mongos_after_restore_fassert.log
          44 kB

        Activity

          People

            kaloian.manassiev@mongodb.com Kaloian Manassiev
            louisa.berger@mongodb.com Louisa Berger
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: