[SERVER-18853] aborting after invariant() failure Created: 06/Jun/15  Updated: 22/Apr/19  Resolved: 06/Jun/15

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Karthik Vamsi Tadepalli Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: OS X
Steps To Reproduce:
  • Download MongoDB tar ball on Mac.
  • Unzip the tar ball using "tar xvf mongodb-osx-x86_64-3.0.3.tgz"
  • Enter mongodb-osx-x86_64-3.0.3
  • Navigate to bin directory of mongodb-osx-x86_64-3.0.3.
  • Become root and create /data/db directories
  • Chmod /data/db to give those directories required permissions to read and write.
  • Exit root and run mongod using the following command "./mongod"
  • Notice that mongo server crashes with the above mentioned log message
Participants:

 Description   

Mongo server crashes upon launch with the following message "aborting after invariant() failure"

Following is the log generated up on running mongo server.

2015-06-06T14:46:57.872+0530 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] journal dir=/data/db/journal
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] recover begin
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] recover lsn: 0
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] recover /data/db/journal/j._0
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] recover cleaning up
2015-06-06T14:46:57.880+0530 I JOURNAL  [initandlisten] removeJournalFiles
2015-06-06T14:46:57.881+0530 I JOURNAL  [initandlisten] recover done
2015-06-06T14:46:57.896+0530 I JOURNAL  [durability] Durability thread started
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] MongoDB starting : pid=2348 port=27017 dbpath=/data/db 64-bit host=CrazyFuck.local
2015-06-06T14:46:57.897+0530 I JOURNAL  [journal writer] Journal writer thread started
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] 
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] db version v3.0.3
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] git version: b40106b36eecd1b4407eb1ad1af6bc60593c6105
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] build info: Darwin bs-osx108-7 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] allocator: system
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] options: {}
2015-06-06T14:46:57.897+0530 I CONTROL  [initandlisten] couldn't open /data/db/local.ns errno:13 Permission denied
2015-06-06T14:46:57.897+0530 F INDEX    [initandlisten] error couldn't open file /data/db/local.ns terminating
2015-06-06T14:46:57.897+0530 I -        [initandlisten] Invariant failure false src/mongo/db/storage/mmap_v1/catalog/namespace_index.cpp 249
2015-06-06T14:46:57.900+0530 I CONTROL  [initandlisten] 
 0x1025a8849 0x10255ad10 0x102547ee3 0x10236e27c 0x10238fb9d 0x102398502 0x101f4e931 0x101e3464c 0x101e32dd3 0x101e37d2c 0x101e32db4
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"101E32000","o":"776849"},{"b":"101E32000","o":"728D10"},{"b":"101E32000","o":"715EE3"},{"b":"101E32000","o":"53C27C"},{"b":"101E32000","o":"55DB9D"},{"b":"101E32000","o":"566502"},{"b":"101E32000","o":"11C931"},{"b":"101E32000","o":"264C"},{"b":"101E32000","o":"DD3"},{"b":"101E32000","o":"5D2C"},{"b":"101E32000","o":"DB4"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Darwin", "release" : "14.3.0", "version" : "Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Karthik/BitAxiom/MongoDBTraining/mongodb-osx-x86_64-3.0.3/bin/./mongod", "machType" : 2, "b" : "101E32000", "buildId" : "A7C3430097EE3D84AB3A80C1DE6D6F1E" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8C5BC000", "buildId" : "CCEC13A5D0D931C5B0B01C564B4A20A6" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8FBA5000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF89479000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8225D000", "buildId" : "D381EBC669D831D380845A80A32CB748" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF84DD8000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF89B4E000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8AEAF000", "buildId" : "E178980139853949B7366B3378873301" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8DE36000", "buildId" : "502CF32B669B3709886208188225E4F0" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8DADA000", "buildId" : "9EACCA38291D38CC811F7E9D1451E2D3" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8484D000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF894C3000", "buildId" : "FA89A113696E32718FE1A0D7324E8481" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8C526000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF81CB2000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF847A3000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF894CF000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF85B8F000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8EBB6000", "buildId" : "86FBED7AF2C83591AD6F486DD57E6B6A" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF89B01000", "buildId" : "20F3B077179D3CB0A3C1C8602D53B4DB" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF87C88000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8F6DD000", "buildId" : "6084A531252339F8B030811FA1A32FB5" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF87C7F000", "buildId" : "62B70ECAE40D3C63896E7F00EC386DDB" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF84824000", "buildId" : "AEB3FE62476330508352D6F9AF961AE6" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8C577000", "buildId" : "EAFD7BD00C303E7D9528F9916BA0167C" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF870BA000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF816BA000", "buildId" : "19BCC25757173502A71F95D65AFA861B" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8C52C000", "buildId" : "589A5F67BE2A3245A1810ECC9B53EB00" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF81429000", "buildId" : "29AB225BD7FB30ED960065D44B9A9442" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8F6D3000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF8BEBB000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF894C4000", "buildId" : "3103AA7F3BAE3673964947FFD7E15C97" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF807AE000", "buildId" : "4CF771286BE03958B646707FA9CE61B2" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF8EC93000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF8F989000", "buildId" : "FBC3F80FA0FB3BD69A7E800DE45F092E" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF8A489000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8ABC6000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF8C9BE000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8B609000", "buildId" : "D35D0DB2D7BD3BE48378062BFE545E1D" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8718A000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF84CBA000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF80D4A000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8839A000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1025a8849]
 mongod(_ZN5mongo10logContextEPKc+0x100) [0x10255ad10]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x183) [0x102547ee3]
 mongod(_ZN5mongo14NamespaceIndex4initEPNS_16OperationContextE+0xAEC) [0x10236e27c]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC2EPNS_16OperationContextERKNS_10StringDataES5_bb+0x14D) [0x10238fb9d]
 mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1C2) [0x102398502]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0xF1) [0x101f4e931]
 mongod(_ZN5mongoL14_initAndListenEi+0x132C) [0x101e3464c]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0x101e32dd3]
 mongod(main+0x3BC) [0x101e37d2c]
 mongod(start+0x34) [0x101e32db4]
-----  END BACKTRACE  -----
2015-06-06T14:46:57.901+0530 I -        [initandlisten] 
 
***aborting after invariant() failure



 Comments   
Comment by Michael Brenden [ 22/Apr/19 ]

Don't forget to check the obvious – file limits

ulimit -n

Default file limit of 1,024 is probably nowhere near enough.

Comment by Ramon Fernandez Marina [ 06/Jun/15 ]

karthikvamsi, please make sure the user you're running mongod under has sufficient permissions to read and write in the /data/db directory, and also read and write all the files within it.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group.

Regards,
Ramón

Comment by Eliot Horowitz (Inactive) [ 06/Jun/15 ]

Note

2015-06-06T14:46:57.897+0530 I CONTROL [initandlisten] couldn't open /data/db/local.ns errno:13 Permission denied
2015-06-06T14:46:57.897+0530 F INDEX [initandlisten] error couldn't open file /data/db/local.ns terminating

Seems the user you are starting mongod as can't access the data files.

Generated at Thu Feb 08 03:48:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.