[SERVER-25067] 3.2.4 with zlib compression: Crash - Fatal Assertion 28558,28559 Created: 14/Jul/16  Updated: 25/Jul/16  Resolved: 25/Jul/16

Status: Closed
Project: Core Server
Component/s: Storage, WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Gavan Acton Assignee: Kelsey Schubert
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We recently had this crash on v3.2.4 using zlib compression. Environment specs & stacks below.

— ENV

2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] MongoDB starting : pid=44128 port=27017 dbpath=/var/lib/mongodb 64-bit host=war-datamine7
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] db version v3.2.4
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] git version: e2ee9ffcf9f5a94fad76802e28cc978718bb7a30
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] allocator: tcmalloc
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] modules: none
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] build environment:
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten]     distmod: ubuntu1404
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten]     distarch: x86_64
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten]     target_arch: x86_64
2016-07-14T13:03:38.486-0400 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", replication: { replSetName: "rsDatalake" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "zlib" }, engineConfig: { directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }

------- Stack Trace 28558

2016-07-14T06:21:18.784-0400 E STORAGE  [repl writer worker 5] WiredTiger (0) [1468491678:783837][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: read checksum error for 4096B block at offset 17095036928: block header checksum of 3484947539 doesn't match expected checksum of 1241204689
2016-07-14T06:21:18.784-0400 E STORAGE  [repl writer worker 5] WiredTiger (0) [1468491678:784337][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: db_logs/index/11--1216780978238540517.wt: encountered an illegal file format or internal value
2016-07-14T06:21:18.784-0400 E STORAGE  [repl writer worker 5] WiredTiger (-31804) [1468491678:784353][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-07-14T06:21:18.784-0400 I -        [repl writer worker 5] Fatal Assertion 28558
2016-07-14T06:21:18.784-0400 I -        [repl writer worker 5]
 
***aborting after fassert() failure
 
 
2016-07-14T06:21:18.819-0400 F -        [repl writer worker 5] Got signal: 6 (Aborted).
 
 0x12f3502 0x12f2659 0x12f2e62 0x7f926ce67330 0x7f926cac8c37 0x7f926cacc028 0x127d9d2 0x1078ef3 0x1a378ec 0x1a37a8d 0x1a37e74 0x1971b16 0x198e7fa 0x1993d90 0x19b3be8 0x198229f 0x19cf28e 0x105696a 0x105a027 0xc91a0f 0xae4fc9 0xae936f 0xae9410 0xac93f5 0xac9761 0xac990b 0xe99c51 0xf2b58e 0xf253e0 0xf27026 0xf27c26 0xf2aedb 0x1288051 0x12889b9 0x1289510 0x7f926d642a60 0x7f926ce5f184 0x7f926cb8c37d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EF3502","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EF2659"},{"b":"400000","o":"EF2E62"},{"b":"7F926CE57000","o":"10330"},{"b":"7F926CA92000","o":"36C37","s":"gsignal"},{"b":"7F926CA92000","o":"3A028","s":"abort"},{"b":"400000","o":"E7D9D2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C78EF3"},{"b":"400000","o":"16378EC","s":"__wt_eventv"},{"b":"400000","o":"1637A8D","s":"__wt_err"},{"b":"400000","o":"1637E74","s":"__wt_panic"},{"b":"400000","o":"1571B16","s":"__wt_bm_read"},{"b":"400000","o":"158E7FA","s":"__wt_bt_read"},{"b":"400000","o":"1593D90","s":"__wt_page_in_func"},{"b":"400000","o":"15B3BE8","s":"__wt_row_search"},{"b":"400000","o":"158229F","s":"__wt_btcur_insert"},{"b":"400000","o":"15CF28E"},{"b":"400000","o":"C5696A","s":"_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"C5A027","s":"_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"891A0F","s":"_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl"},{"b":"400000","o":"6E4FC9","s":"_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},{"b":"400000","o":"6E936F","s":"_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},{"b":"400000","o":"6E9410","s":"_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE"},{"b":"400000","o":"6C93F5","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b"},{"b":"400000","o":"6C9761","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb"},{"b":"400000","o":"6C990B","s":"_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb"},{"b":"400000","o":"A99C51","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb"},{"b":"400000","o":"B2B58E","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b"},{"b":"400000","o":"B253E0"},{"b":"400000","o":"B27026","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE"},{"b":"400000","o":"B27C26","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"400000","o":"B2AEDB","s":"_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE"},{"b":"400000","o":"E88051","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"400000","o":"E889B9","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"400000","o":"E89510","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs"},{"b":"7F926D591000","o":"B1A60"},{"b":"7F926CE57000","o":"8184"},{"b":"7F926CA92000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.4", "gitVersion" : "e2ee9ffcf9f5a94fad76802e28cc978718bb7a30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-34-generic", "version" : "#39~14.04.1-Ubuntu SMP Fri Mar 11 11:38:02 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EF46210F8976780D45B811C3540FECB9E734EABE" }, { "b" : "7FFDFC9A0000", "elfType" : 3, "buildId" : "89FBCFEB3FFCA6A9AAB57384D5EC6D1C8AB68009" }, { "b" : "7F926E07D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F926DCA1000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F926DA99000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F926D895000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F926D591000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "76190E922AF7457D078F75C9B15FA184E83EB506" }, { "b" : "7F926D28B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F926D075000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F926CE57000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F926CA92000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F926E2DC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f3502]
 mongod(+0xEF2659) [0x12f2659]
 mongod(+0xEF2E62) [0x12f2e62]
 libpthread.so.0(+0x10330) [0x7f926ce67330]
 libc.so.6(gsignal+0x37) [0x7f926cac8c37]
 libc.so.6(abort+0x148) [0x7f926cacc028]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127d9d2]
 mongod(+0xC78EF3) [0x1078ef3]
 mongod(__wt_eventv+0x40C) [0x1a378ec]
 mongod(__wt_err+0x8D) [0x1a37a8d]
 mongod(__wt_panic+0x24) [0x1a37e74]
 mongod(__wt_bm_read+0x76) [0x1971b16]
 mongod(__wt_bt_read+0x1EA) [0x198e7fa]
 mongod(__wt_page_in_func+0x180) [0x1993d90]
 mongod(__wt_row_search+0x698) [0x19b3be8]
 mongod(__wt_btcur_insert+0x45F) [0x198229f]
 mongod(+0x15CF28E) [0x19cf28e]
 mongod(_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb+0x12A) [0x105696a]
 mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0xC7) [0x105a027]
 mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x19F) [0xc91a0f]
 mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x109) [0xae4fc9]
 mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x11F) [0xae936f]
 mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE+0x80) [0xae9410]
 mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x325) [0xac93f5]
 mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xac9761]
 mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb+0x6B) [0xac990b]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0xF41) [0xe99c51]
 mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf2b58e]
 mongod(+0xB253E0) [0xf253e0]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0x
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf27c26]
 mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf2aedb]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x1288051]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12889b9]
 mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x1289510]
 libstdc++.so.6(+0xB1A60) [0x7f926d642a60]
 libpthread.so.0(+0x8184) [0x7f926ce5f184]
 libc.so.6(clone+0x6D) [0x7f926cb8c37d]
-----  END BACKTRACE  -----

Booting up server after crash produces

------- Stack Trace 28559

2016-07-14T13:03:53.379-0400 E STORAGE  [repl writer worker 8] WiredTiger (0) [1468515833:379294][44128:0x7ff178aa1700]
2016-07-14T13:03:53.379-0400 E STORAGE  [repl writer worker 8] WiredTiger (0) [1468515833:379422][44128:0x7ff178aa1700]
2016-07-14T13:03:53.379-0400 E STORAGE  [repl writer worker 8] WiredTiger (-31804) [1468515833:379455][44128:0x7ff178aa1700]
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 4] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 13] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 9] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 2] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 9]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 2]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 8] Fatal Assertion 28558
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 11] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 12] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 3] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 3]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 13]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 14] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 14]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [rsSync] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [rsSync]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 11]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 4]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 12]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 6] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 6]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 0] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 0]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 7] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 8]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 7]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 5] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 15] Fatal Assertion 28559
2016-07-14T13:03:53.379-0400 I -        [repl writer worker 10] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 15]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 10]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 1] Fatal Assertion 28559
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 1]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.380-0400 I -        [repl writer worker 5]
 
***aborting after fassert() failure
 
 
2016-07-14T13:03:53.397-0400 F -        [repl writer worker 9] Got signal: 6 (Aborted).
 
 0x12f3502 0x12f2659 0x12f2e62 0x7ff18c4c9330 0x7ff18c12ac37 0x7ff18c12e028 0x127d9d2 0x1079345 0x106e3a7 0xc13089 0xbdaffa 0xc0ff04 0xe0ac85 0xe0b349 0xe0b445 0xd1d479 0xe9a1dc 0xf2b58e 0xf253e0 0xf27026 0xf27c26 0xf2aedb 0x1288051 0x12889b9 0x1289510 0x7ff18cca4a60 0x7ff18c4c1184 0x7ff18c1ee37d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000"
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f3502]
 mongod(+0xEF2659) [0x12f2659]
 mongod(+0xEF2E62) [0x12f2e62]
 libpthread.so.0(+0x10330) [0x7ff18c4c9330]
 libc.so.6(gsignal+0x37) [0x7ff18c12ac37]
 libc.so.6(abort+0x148) [0x7ff18c12e028]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127d9d2]
 mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x355) [0x1079345]
 mongod(_ZN5mongo21WiredTigerRecordStore6Cursor9seekExactERKNS_8RecordIdE+0x157) [0x106e3a7]
 mongod(_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEE+0x99) [0xc13089]
 mongod(_ZN5mongo11IDHackStage4workEPm+0x21A) [0xbdaffa]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x394) [0xc0ff04]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe0ac85]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe0b349]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe0b445]
 mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x179) [0xd1d479]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0x14CC) [0xe9a1dc]
 mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf2b58e]
 mongod(+0xB253E0) [0xf253e0]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf27026]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf27c26]
 mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf2aedb]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x1288051]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12889b9]
 mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x1289510]
 libstdc++.so.6(+0xB1A60) [0x7ff18cca4a60]
 libpthread.so.0(+0x8184) [0x7ff18c4c1184]
 libc.so.6(clone+0x6D) [0x7ff18c1ee37d]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 25/Jul/16 ]

Hi GACTON,

Thank you for answering my questions. Unfortunately, I do not see anything in your response that clearly points to a root cause either in your disk system, MongoDB, or system configuration settings.

Since we are unable to reproduce this issue, I am closing this ticket. Please open a new ticket if this issue occurs again.

Kind regards,
Thomas

Comment by Gavan Acton [ 14/Jul/16 ]

1. No other errors. A number of 'getmore' operations based on running queries.
2. Locally attached device - Massive 8TB HDD. Redundancy achieved via cluster replication. Volume management: /dev/sdb1 /var/lib/mongodb ext4 rw,errors=remount-ro 0 0
3. No.
4. No
5. No need for backups. The data added into this cluster is from backup archives.

Thanks!

Comment by Kelsey Schubert [ 14/Jul/16 ]

Hi GACTON,

These error messages indicate some data files have become corrupt in some way. I've assembled a list of routine questions about data storage and the configuration of your environment, which will help us examine this issue. However, please note that it may be challenging to identify the root cause of the the corruption without a straightforward reproduction.

  1. Around this operation, were there any other server errors logged?
  2. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  3. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?
  4. Have you ever restored this instance from backups?
  5. What method do you use to create backups?

I would recommend a clean resync from a node that is not affected.

Thank you,
Thomas

Generated at Thu Feb 08 04:08:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.