[SERVER-26901] Invariant failure on startup when upgrading from 3.0.3 Created: 03/Nov/16  Updated: 04/Nov/16  Resolved: 04/Nov/16

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: 3.0.13
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Sumeet Patel Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian (Jessie)


Operating System: ALL
Steps To Reproduce:

Upgrade from 3.0.3 to 3.0.13 on Debian Jessie which has been upgraded from Squeeze (Mongo 2.2.6)

The blocking error on startup was:
Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187

I guess the error reason could have been more informative at the least.

Jumping over 3.0.13 to 3.2.10 didn't resolve the issue.

My practical solution was to purge the data files and then re-sync up to my primary running 3.0.3, moving from mmapv1 to wiredtiger as the storage engine, then swap roles and upgrading and so forth.

Apologies for the missing top of the log.

mongod.conf

# mongod.conf
 
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/
 
# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:
 
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
 
# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1
 
 
#processManagement:
 
#security:
 
#operationProfiling:
 
#replication:
 
#sharding:
 
## Enterprise-Only Options:
 
#auditLog:
 
#snmp:

mongod.log

2016-11-03T15:40:31.474+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/journal
2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover begin
2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover lsn: 0
2016-11-03T15:40:31.491+0000 I JOURNAL  [initandlisten] recover /var/lib/mongodb/journal/j._0
2016-11-03T15:40:31.509+0000 I JOURNAL  [initandlisten] recover cleaning up
2016-11-03T15:40:31.509+0000 I JOURNAL  [initandlisten] removeJournalFiles
2016-11-03T15:40:31.616+0000 I JOURNAL  [initandlisten] recover done
2016-11-03T15:40:31.617+0000 I JOURNAL  [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2016-11-03T15:40:34.016+0000 I -        [initandlisten]   File Preallocator Progress: 817889280/1073741824 76%
2016-11-03T15:40:37.193+0000 I -        [initandlisten]   File Preallocator Progress: 975175680/1073741824 90%
2016-11-03T15:40:54.365+0000 I JOURNAL  [durability] Durability thread started
2016-11-03T15:40:54.365+0000 I JOURNAL  [journal writer] Journal writer thread started
2016-11-03T15:40:54.372+0000 I -        [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187
2016-11-03T15:40:54.372+0000 I -        [initandlisten]
 
***aborting after invariant() failure
 
 
2016-11-03T15:40:54.379+0000 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x13287d2 0x1327909 0x1328112 0x7f2b20f138d0 0x7f2b20b8e067 0x7f2b20b8f448 0x12ae47b 0xafe7fd 0xb00324 0xb031cd 0x9b7c8a 0x9b9750 0x97026d 0x7f2b20b7ab45 0x9b3a97
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F287D2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F27909"},{"b":"400000","o":"F28112"},{"b":"7F2B20F04000","o":"F8D0"},{"b":"7F2B20B59000","o":"35067","s":"gsignal"},{"b":"7F2B20B59000","o":"36448","s":"abort"},{"b":"400000","o":"EAE47B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"6FE7FD","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"700324","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"7031CD","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5B7C8A"},{"b":"400000","o":"5B9750","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57026D","s":"main"},{"b":"7F2B20B59000","o":"21B45","s":"__libc_start_main"},{"b":"400000","o":"5B3A97"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03AADFF60B81BAF2675CA42BB00380AB31F376AF" }, { "b" : "7FFE04737000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F3A70B41FDF412571F8C6914BB9907286C4D65C8" }, { "b" : "7F2B21E40000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F2B21A44000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F2B2183C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "18650C365FBA12BA12DB680B988B7754C0F99364" }, { "b" : "7F2B21638000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1D841026FF2C5AEA25C21921FAC9C978AB465825" }, { "b" : "7F2B21337000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BBA5CF06072F15BEA36FC8B0A2C6A32BBF08C235" }, { "b" : "7F2B21121000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F2B20F04000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B9DD2674891C5395C5724321158DC2EB8C7DA64B" }, { "b" : "7F2B20B59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AD0CE0A6B6E201C9A8195B2D55043AFCD4F5DB00" }, { "b" : "7F2B220A1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "44DA6E33EBAB324C1D0128F9465425AD355CC978" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13287d2]
 mongod(+0xF27909) [0x1327909]
 mongod(+0xF28112) [0x1328112]
 libpthread.so.0(+0xF8D0) [0x7f2b20f138d0]
 libc.so.6(gsignal+0x37) [0x7f2b20b8e067]
 libc.so.6(abort+0x148) [0x7f2b20b8f448]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12ae47b]
 mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xCD) [0xafe7fd]
 mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb00324]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xb031cd]
 mongod(+0x5B7C8A) [0x9b7c8a]
 mongod(_ZN5mongo13initAndListenEi+0x10) [0x9b9750]
 mongod(main+0x15D) [0x97026d]
 libc.so.6(__libc_start_main+0xF5) [0x7f2b20b7ab45]
 mongod(+0x5B3A97) [0x9b3a97]
-----  END BACKTRACE  -----

Participants:

 Description   

I was upgrading an environment, Debian (Squeeze) running Mongodb 2.2.6.
I upgraded to Wheezy, then Jessie, then proceeded to upgrade my replica set to 2.4 then 2.6 then to 3.0.13, then my Mongod stopped starting up.



 Comments   
Comment by Ramon Fernandez Marina [ 04/Nov/16 ]

The error message:

2016-11-03T15:40:54.372+0000 I -        [initandlisten] Invariant failure cce.get() src/mongo/db/catalog/database.cpp 187

is indicative of data corruption in this node, most typically caused by unreliable storage. The right solution is to resync from a healthy node, as you already did.

I'm therefore closing this ticket. In the future, please post on the mongodb-user group or Stack Overflow with the mongodb tag for MongoDB-related support requests, where your question will reach a larger audience.

Thanks,
Ramón.

Generated at Thu Feb 08 04:13:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.