Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26901

Invariant failure on startup when upgrading from 3.0.3

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor - P4 Minor - P4
    • None
    • 3.0.13
    • Shell
    • None
    • Debian (Jessie)
    • ALL
    • Hide

      Upgrade from 3.0.3 to 3.0.13 on Debian Jessie which has been upgraded from Squeeze (Mongo 2.2.6)

      The blocking error on startup was:
      Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187

      I guess the error reason could have been more informative at the least.

      Jumping over 3.0.13 to 3.2.10 didn't resolve the issue.

      My practical solution was to purge the data files and then re-sync up to my primary running 3.0.3, moving from mmapv1 to wiredtiger as the storage engine, then swap roles and upgrading and so forth.

      Apologies for the missing top of the log.

      mongod.conf

      # mongod.conf
       
      # for documentation of all options, see:
      #   http://docs.mongodb.org/manual/reference/configuration-options/
       
      # Where and how to store data.
      storage:
        dbPath: /var/lib/mongodb
        journal:
          enabled: true
      #  engine:
      #  mmapv1:
      #  wiredTiger:
       
      # where to write logging data.
      systemLog:
        destination: file
        logAppend: true
        path: /var/log/mongodb/mongod.log
       
      # network interfaces
      net:
        port: 27017
        bindIp: 127.0.0.1
       
       
      #processManagement:
       
      #security:
       
      #operationProfiling:
       
      #replication:
       
      #sharding:
       
      ## Enterprise-Only Options:
       
      #auditLog:
       
      #snmp:
      

      mongod.log

      2016-11-03T15:40:31.474+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
      2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/journal
      2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover begin
      2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
      2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover lsn: 0
      2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover /var/lib/mongodb/journal/j._0
      2016-11-03T15:40:31.509+0000 I JOURNAL  [initandlisten] recover cleaning up
      2016-11-03T15:40:31.509+0000 I JOURNAL  [initandlisten] removeJournalFiles
      2016-11-03T15:40:31.616+0000 I JOURNAL  [initandlisten] recover done
      2016-11-03T15:40:31.617+0000 I JOURNAL  [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
      2016-11-03T15:40:34.016+0000 I -        [initandlisten]   File Preallocator Progress: 817889280/1073741824 76%
      2016-11-03T15:40:37.193+0000 I -        [initandlisten]   File Preallocator Progress: 975175680/1073741824 90%
      2016-11-03T15:40:54.365+0000 I JOURNAL  [durability] Durability thread started
      2016-11-03T15:40:54.365+0000 I JOURNAL  [journal writer] Journal writer thread started
      2016-11-03T15:40:54.372+0000 I -        [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187
      2016-11-03T15:40:54.372+0000 I -        [initandlisten]
       
      ***aborting after invariant() failure
       
       
      2016-11-03T15:40:54.379+0000 F -        [initandlisten] Got signal: 6 (Aborted).
       
       0x13287d2 0x1327909 0x1328112 0x7f2b20f138d0 0x7f2b20b8e067 0x7f2b20b8f448 0x12ae47b 0xafe7fd 0xb00324 0xb031cd 0x9b7c8a 0x9b9750 0x97026d 0x7f2b20b7ab45 0x9b3a97
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F287D2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27909"},{"b":"400000","o":"F28112"},{"b":"7F2B20F04000","o":"F8D0"},{"b":"7F2B20B59000","o":"35067","s":"gsignal"},{"b":"7F2B20B59000","o":"36448","s":"abort"},{"b":"400000","o":"EAE47B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"6FE7FD","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"700324","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"7031CD","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5B7C8A"},{"b":"400000","o":"5B9750","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57026D","s":"main"},{"b":"7F2B20B59000","o":"21B45","s":"__libc_start_main"},{"b":"400000","o":"5B3A97"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03AADFF60B81BAF2675CA42BB00380AB31F376AF" }, { "b" : "7FFE04737000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F3A70B41FDF412571F8C6914BB9907286C4D65C8" }, { "b" : "7F2B21E40000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F2B21A44000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F2B2183C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F2B21638000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F2B21337000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F2B21121000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F2B20F04000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F2B20B59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F2B220A1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13287d2]
       mongod(+0xF27909) [0x1327909]
       mongod(+0xF28112) [0x1328112]
       libpthread.so.0(+0xF8D0) [0x7f2b20f138d0]
       libc.so.6(gsignal+0x37) [0x7f2b20b8e067]
       libc.so.6(abort+0x148) [0x7f2b20b8f448]
       mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12ae47b]
       mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xCD) [0xafe7fd]
       mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb00324]
       mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xb031cd]
       mongod(+0x5B7C8A) [0x9b7c8a]
       mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b9750]
       mongod(main+0x15D) [0x97026d]
       libc.so.6(__libc_start_main+0xF5) [0x7f2b20b7ab45]
       mongod(+0x5B3A97) [0x9b3a97]
      -----  END BACKTRACE  -----
      

      Show
      Upgrade from 3.0.3 to 3.0.13 on Debian Jessie which has been upgraded from Squeeze (Mongo 2.2.6) The blocking error on startup was: Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187 I guess the error reason could have been more informative at the least. Jumping over 3.0.13 to 3.2.10 didn't resolve the issue. My practical solution was to purge the data files and then re-sync up to my primary running 3.0.3, moving from mmapv1 to wiredtiger as the storage engine, then swap roles and upgrading and so forth. Apologies for the missing top of the log. mongod.conf # mongod.conf # for documentation of all options, see: # http://docs.mongodb.org/manual/reference/configuration-options/ # Where and how to store data. storage: dbPath: /var/lib/mongodb journal: enabled: true # engine: # mmapv1: # wiredTiger: # where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log # network interfaces net: port: 27017 bindIp: 127.0.0.1 #processManagement: #security: #operationProfiling: #replication: #sharding: ## Enterprise-Only Options: #auditLog: #snmp: mongod.log 2016-11-03T15:40:31.474+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover begin 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] info no lsn file in journal/ directory 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover lsn: 0 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._0 2016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] recover cleaning up 2016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] removeJournalFiles 2016-11-03T15:40:31.616+0000 I JOURNAL [initandlisten] recover done 2016-11-03T15:40:31.617+0000 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0 2016-11-03T15:40:34.016+0000 I - [initandlisten] File Preallocator Progress: 817889280/1073741824 76% 2016-11-03T15:40:37.193+0000 I - [initandlisten] File Preallocator Progress: 975175680/1073741824 90% 2016-11-03T15:40:54.365+0000 I JOURNAL [durability] Durability thread started 2016-11-03T15:40:54.365+0000 I JOURNAL [journal writer] Journal writer thread started 2016-11-03T15:40:54.372+0000 I - [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187 2016-11-03T15:40:54.372+0000 I - [initandlisten] ***aborting after invariant() failure 2016-11-03T15:40:54.379+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x13287d2 0x1327909 0x1328112 0x7f2b20f138d0 0x7f2b20b8e067 0x7f2b20b8f448 0x12ae47b 0xafe7fd 0xb00324 0xb031cd 0x9b7c8a 0x9b9750 0x97026d 0x7f2b20b7ab45 0x9b3a97 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F287D2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27909"},{"b":"400000","o":"F28112"},{"b":"7F2B20F04000","o":"F8D0"},{"b":"7F2B20B59000","o":"35067","s":"gsignal"},{"b":"7F2B20B59000","o":"36448","s":"abort"},{"b":"400000","o":"EAE47B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"6FE7FD","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"700324","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"7031CD","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5B7C8A"},{"b":"400000","o":"5B9750","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57026D","s":"main"},{"b":"7F2B20B59000","o":"21B45","s":"__libc_start_main"},{"b":"400000","o":"5B3A97"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03AADFF60B81BAF2675CA42BB00380AB31F376AF" }, { "b" : "7FFE04737000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F3A70B41FDF412571F8C6914BB9907286C4D65C8" }, { "b" : "7F2B21E40000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F2B21A44000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F2B2183C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F2B21638000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F2B21337000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F2B21121000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F2B20F04000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F2B20B59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F2B220A1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13287d2] mongod(+0xF27909) [0x1327909] mongod(+0xF28112) [0x1328112] libpthread.so.0(+0xF8D0) [0x7f2b20f138d0] libc.so.6(gsignal+0x37) [0x7f2b20b8e067] libc.so.6(abort+0x148) [0x7f2b20b8f448] mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12ae47b] mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xCD) [0xafe7fd] mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb00324] mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xb031cd] mongod(+0x5B7C8A) [0x9b7c8a] mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b9750] mongod(main+0x15D) [0x97026d] libc.so.6(__libc_start_main+0xF5) [0x7f2b20b7ab45] mongod(+0x5B3A97) [0x9b3a97] ----- END BACKTRACE -----

    Description

      I was upgrading an environment, Debian (Squeeze) running Mongodb 2.2.6.
      I upgraded to Wheezy, then Jessie, then proceeded to upgrade my replica set to 2.4 then 2.6 then to 3.0.13, then my Mongod stopped starting up.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sumeetkpatel Sumeet Patel
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: