[SERVER-31813] mongod crashes repeatedly Created: 03/Nov/17  Updated: 06/Aug/18  Resolved: 01/Dec/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.12
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Sandesh Kumar Shetty Assignee: Mark Agarunov
Resolution: Incomplete Votes: 0
Labels: envns, rns, rpns, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongodb.log    
Operating System: Linux
Steps To Reproduce:

NA

Participants:

 Description   

Hi Team,

mongod is crashing continously with the below logs. This is happening repeatedly because of which mongod doesnt come up.

This is on linux platform.

2017-10-13T16:59:49.436+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7489 port=27017 dbpath=/var/opt/lamplighter/data 64-bit host=sojo-cfmcv-01.provocraft.com
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] db version v3.2.12
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] modules: none
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] build environment:
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] options: { config: "/var/opt/lamplighter/etc/mongod.conf", storage: { dbPath: "/var/opt/lamplighter/data", engine: "wiredTiger", wiredTiger: { engineConfig: { configString: "cache_size=1536M" } } }, systemLog: { component: { accessControl: { verbosity: 0 }, command: { verbosity: 0 } }, destination: "file", logAppend: true, logRotate: "reopen", path: "/var/opt/lamplighter/log/mongodb.log", timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } }
2017-10-13T16:59:49.495+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=1536M
2017-10-13T16:59:49.741+0000 E STORAGE  [initandlisten] WiredTiger (-31802) [1507913989:740996][7489:0x7f4165a00c80], file:collection-40--8734011645000920999.wt, WT_SESSION.open_cursor: unable to read root page from file:collection-40--8734011645000920999.wt: WT_ERROR: non-specific WiredTiger error
2017-10-13T16:59:49.741+0000 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 79
2017-10-13T16:59:49.751+0000 I CONTROL  [initandlisten]
 0x130ff72 0x12bae28 0x12a314d 0x108d304 0x108bc00 0x1087c22 0x108662d 0x10771db 0xfd66d4 0xfdc472 0x1075a58 0xf996e8 0x95a91e 0x95cec0 0x9600cd 0x7f4164930635 0x9572d9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F0FF72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EBAE28","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"EA314D","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"C8D304","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C8BC00","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C87C22","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C8662D","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C771DB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BD66D4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BDC472","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C75A58"},{"b":"400000","o":"B996E8","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"55A91E"},{"b":"400000","o":"55CEC0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5600CD","s":"main"},{"b":"7F416490F000","o":"21635","s":"__libc_start_main"},{"b":"400000","o":"5572D9"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.53sf.virtual-150", "version" : "#1 SMP Thu Aug 31 19:10:36 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFEEEFE000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7F41655E7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F41653E3000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F41650E9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F4164ED4000", "path" : "/usr/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4164CB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F416490F000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F41657F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ff72]
 mongod(_ZN5mongo10logContextEPKc+0x138) [0x12bae28]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xAD) [0x12a314d]
 mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0xE4) [0x108d304]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x108bc00]
 mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x1087c22]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x3ED) [0x108662d]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x1AB) [0x10771db]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfd66d4]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x6D2) [0xfdc472]
 mongod(+0xC75A58) [0x1075a58]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf996e8]
 mongod(+0x55A91E) [0x95a91e]
 mongod(_ZN5mongo13initAndListenEi+0x10) [0x95cec0]
 mongod(main+0x15D) [0x9600cd]
 libc.so.6(__libc_start_main+0xF5) [0x7f4164930635]
 mongod(+0x5572D9) [0x9572d9]
-----  END BACKTRACE  -----
2017-10-13T16:59:49.751+0000 I -        [initandlisten]
 
***aborting after invariant() failure

Please let us know if there is any workaround.
Can I run "mongod —repair" here? Would it help?

Thanks,
Sandesh



 Comments   
Comment by Mark Agarunov [ 01/Dec/17 ]

Hello sandshet,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide the WiredTiger.wt and WiredTiger.turtle files and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 06/Nov/17 ]

Hello sandshet,

Thank you for the report. Please provide the WiredTiger.wt and WiredTiger.turtle files so that we can attempt a repair of the database. However, please keep in mind that this is not a guaranteed fix.

Thanks,
Mark

Generated at Thu Feb 08 04:28:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.