Details
-
Bug
-
Resolution: Done
-
Minor - P4
-
None
-
3.0.13
-
None
-
Debian (Jessie)
-
ALL
-
Hide
Upgrade from 3.0.3 to 3.0.13 on Debian Jessie which has been upgraded from Squeeze (Mongo 2.2.6)
The blocking error on startup was:
Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187I guess the error reason could have been more informative at the least.
Jumping over 3.0.13 to 3.2.10 didn't resolve the issue.
My practical solution was to purge the data files and then re-sync up to my primary running 3.0.3, moving from mmapv1 to wiredtiger as the storage engine, then swap roles and upgrading and so forth.
Apologies for the missing top of the log.
mongod.conf
# mongod.conf# for documentation of all options, see:# http://docs.mongodb.org/manual/reference/configuration-options/# Where and how to store data.storage:dbPath: /var/lib/mongodbjournal:enabled: true# engine:# mmapv1:# wiredTiger:# where to write logging data.systemLog:destination: filelogAppend: truepath: /var/log/mongodb/mongod.log# network interfacesnet:port: 27017bindIp: 127.0.0.1#processManagement:#security:#operationProfiling:#replication:#sharding:## Enterprise-Only Options:#auditLog:#snmp:mongod.log
2016-11-03T15:40:31.474+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover begin2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] info no lsn file in journal/ directory2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover lsn: 02016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._02016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] recover cleaning up2016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] removeJournalFiles2016-11-03T15:40:31.616+0000 I JOURNAL [initandlisten] recover done2016-11-03T15:40:31.617+0000 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.02016-11-03T15:40:34.016+0000 I - [initandlisten] File Preallocator Progress: 817889280/1073741824 76%2016-11-03T15:40:37.193+0000 I - [initandlisten] File Preallocator Progress: 975175680/1073741824 90%2016-11-03T15:40:54.365+0000 I JOURNAL [durability] Durability thread started2016-11-03T15:40:54.365+0000 I JOURNAL [journal writer] Journal writer thread started2016-11-03T15:40:54.372+0000 I - [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 1872016-11-03T15:40:54.372+0000 I - [initandlisten]***aborting after invariant() failure2016-11-03T15:40:54.379+0000 F - [initandlisten] Got signal: 6 (Aborted).0x13287d2 0x1327909 0x1328112 0x7f2b20f138d0 0x7f2b20b8e067 0x7f2b20b8f448 0x12ae47b 0xafe7fd 0xb00324 0xb031cd 0x9b7c8a 0x9b9750 0x97026d 0x7f2b20b7ab45 0x9b3a97----- BEGIN BACKTRACE -----{"backtrace":[{"b":"400000","o":"F287D2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27909"},{"b":"400000","o":"F28112"},{"b":"7F2B20F04000","o":"F8D0"},{"b":"7F2B20B59000","o":"35067","s":"gsignal"},{"b":"7F2B20B59000","o":"36448","s":"abort"},{"b":"400000","o":"EAE47B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"6FE7FD","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"700324","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"7031CD","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5B7C8A"},{"b":"400000","o":"5B9750","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57026D","s":"main"},{"b":"7F2B20B59000","o":"21B45","s":"__libc_start_main"},{"b":"400000","o":"5B3A97"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03AADFF60B81BAF2675CA42BB00380AB31F376AF" }, { "b" : "7FFE04737000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F3A70B41FDF412571F8C6914BB9907286C4D65C8" }, { "b" : "7F2B21E40000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F2B21A44000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F2B2183C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F2B21638000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F2B21337000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F2B21121000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F2B20F04000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F2B20B59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F2B220A1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }}mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13287d2]mongod(+0xF27909) [0x1327909]mongod(+0xF28112) [0x1328112]libpthread.so.0(+0xF8D0) [0x7f2b20f138d0]libc.so.6(gsignal+0x37) [0x7f2b20b8e067]libc.so.6(abort+0x148) [0x7f2b20b8f448]mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12ae47b]mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xCD) [0xafe7fd]mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb00324]mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xb031cd]mongod(+0x5B7C8A) [0x9b7c8a]mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b9750]mongod(main+0x15D) [0x97026d]libc.so.6(__libc_start_main+0xF5) [0x7f2b20b7ab45]mongod(+0x5B3A97) [0x9b3a97]----- END BACKTRACE -----ShowUpgrade from 3.0.3 to 3.0.13 on Debian Jessie which has been upgraded from Squeeze (Mongo 2.2.6) The blocking error on startup was: Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187 I guess the error reason could have been more informative at the least. Jumping over 3.0.13 to 3.2.10 didn't resolve the issue. My practical solution was to purge the data files and then re-sync up to my primary running 3.0.3, moving from mmapv1 to wiredtiger as the storage engine, then swap roles and upgrading and so forth. Apologies for the missing top of the log. mongod.conf # mongod.conf # for documentation of all options, see: # http://docs.mongodb.org/manual/reference/configuration-options/ # Where and how to store data. storage: dbPath: /var/lib/mongodb journal: enabled: true # engine: # mmapv1: # wiredTiger: # where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log # network interfaces net: port: 27017 bindIp: 127.0.0.1 #processManagement: #security: #operationProfiling: #replication: #sharding: ## Enterprise-Only Options: #auditLog: #snmp: mongod.log 2016-11-03T15:40:31.474+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover begin 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] info no lsn file in journal/ directory 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover lsn: 0 2016-11-03T15:40:31.491+0000 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._0 2016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] recover cleaning up 2016-11-03T15:40:31.509+0000 I JOURNAL [initandlisten] removeJournalFiles 2016-11-03T15:40:31.616+0000 I JOURNAL [initandlisten] recover done 2016-11-03T15:40:31.617+0000 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0 2016-11-03T15:40:34.016+0000 I - [initandlisten] File Preallocator Progress: 817889280/1073741824 76% 2016-11-03T15:40:37.193+0000 I - [initandlisten] File Preallocator Progress: 975175680/1073741824 90% 2016-11-03T15:40:54.365+0000 I JOURNAL [durability] Durability thread started 2016-11-03T15:40:54.365+0000 I JOURNAL [journal writer] Journal writer thread started 2016-11-03T15:40:54.372+0000 I - [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187 2016-11-03T15:40:54.372+0000 I - [initandlisten] ***aborting after invariant() failure 2016-11-03T15:40:54.379+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x13287d2 0x1327909 0x1328112 0x7f2b20f138d0 0x7f2b20b8e067 0x7f2b20b8f448 0x12ae47b 0xafe7fd 0xb00324 0xb031cd 0x9b7c8a 0x9b9750 0x97026d 0x7f2b20b7ab45 0x9b3a97 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F287D2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27909"},{"b":"400000","o":"F28112"},{"b":"7F2B20F04000","o":"F8D0"},{"b":"7F2B20B59000","o":"35067","s":"gsignal"},{"b":"7F2B20B59000","o":"36448","s":"abort"},{"b":"400000","o":"EAE47B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"6FE7FD","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"700324","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"7031CD","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5B7C8A"},{"b":"400000","o":"5B9750","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57026D","s":"main"},{"b":"7F2B20B59000","o":"21B45","s":"__libc_start_main"},{"b":"400000","o":"5B3A97"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03AADFF60B81BAF2675CA42BB00380AB31F376AF" }, { "b" : "7FFE04737000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F3A70B41FDF412571F8C6914BB9907286C4D65C8" }, { "b" : "7F2B21E40000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F2B21A44000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F2B2183C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F2B21638000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F2B21337000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F2B21121000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F2B20F04000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F2B20B59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F2B220A1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13287d2] mongod(+0xF27909) [0x1327909] mongod(+0xF28112) [0x1328112] libpthread.so.0(+0xF8D0) [0x7f2b20f138d0] libc.so.6(gsignal+0x37) [0x7f2b20b8e067] libc.so.6(abort+0x148) [0x7f2b20b8f448] mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12ae47b] mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xCD) [0xafe7fd] mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb00324] mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xb031cd] mongod(+0x5B7C8A) [0x9b7c8a] mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b9750] mongod(main+0x15D) [0x97026d] libc.so.6(__libc_start_main+0xF5) [0x7f2b20b7ab45] mongod(+0x5B3A97) [0x9b3a97] ----- END BACKTRACE -----
Description
I was upgrading an environment, Debian (Squeeze) running Mongodb 2.2.6.
I upgraded to Wheezy, then Jessie, then proceeded to upgrade my replica set to 2.4 then 2.6 then to 3.0.13, then my Mongod stopped starting up.