Details
-
Question
-
Resolution: Done
-
Critical - P2
-
None
-
None
-
None
Description
My mongodb which version is 3.0.7 is crashed, the following is detailed information:
2016-06-20T12:43:50.886+0800 E STORAGE [repl writer worker 1] WiredTiger (0) [1466397830:886662][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: read checksum error for 16384B block at offset 7593738240: calculated block checksum of 384598265 doesn't match expected checksum of 2923502613
|
2016-06-20T12:43:50.886+0800 E STORAGE [repl writer worker 1] WiredTiger (0) [1466397830:886748][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: collection-143-1451283047013864403.wt: encountered an illegal file format or internal value
|
2016-06-20T12:43:50.886+0800 E STORAGE [repl writer worker 1] WiredTiger (-31804) [1466397830:886769][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 12] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 7] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 10] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 4] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 1] Fatal Assertion 28558
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 6] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 5] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 0] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 8] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 13] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 15] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 9] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 14] Fatal Assertion 28559
|
2016-06-20T12:43:50.886+0800 I - [repl writer worker 3] Fatal Assertion 28559
|
2016-06-20T12:43:50.887+0800 I - [repl writer worker 11] Fatal Assertion 28559
|
2016-06-20T12:43:50.887+0800 I - [repl writer worker 2] Fatal Assertion 28559
|
2016-06-20T12:43:50.890+0800 I ACCESS [conn108] Successfully authenticated as principal qxbdev on iEnterprise
|
2016-06-20T12:43:50.891+0800 I NETWORK [initandlisten] connection accepted from 10.47.90.78:54524 #110 (56 connections now open)
|
2016-06-20T12:43:50.896+0800 I ACCESS [conn107] Successfully authenticated as principal qxbdev on iEnterprise
|
2016-06-20T12:43:50.898+0800 I NETWORK [conn107] end connection 10.47.90.78:54520 (55 connections now open)
|
2016-06-20T12:43:50.898+0800 I NETWORK [initandlisten] connection accepted from 10.47.90.78:54525 #111 (56 connections now open)
|
2016-06-20T12:43:50.898+0800 I CONTROL [repl writer worker 6]
|
0xf5e979 0xf07861 0xeeb491 0xd886a0 0xd7d134 0xd7d192 0xd7d1d5 0xd6a485 0xa73f15 0x9f5710 0xa2273c 0xbc6464 0xbc6814 0xbc6e7d 0xb123c8 0xc33a78 0xc9bebf 0xc9ed65 0xefdd4b 0xfac904 0x7fd76ae8bdf3 0x7fd769f8e1ad
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B5E979"},{"b":"400000","o":"B07861"},{"b":"400000","o":"AEB491"},{"b":"400000","o":"9886A0"},{"b":"400000","o":"97D134"},{"b":"400000","o":"97D192"},{"b":"400000","o":"97D1D5"},{"b":"400000","o":"96A485"},{"b":"400000","o":"673F15"},{"b":"400000","o":"5F5710"},{"b":"400000","o":"62273C"},{"b":"400000","o":"7C6464"},{"b":"400000","o":"7C6814"},{"b":"400000","o":"7C6E7D"},{"b":"400000","o":"7123C8"},{"b":"400000","o":"833A78"},{"b":"400000","o":"89BEBF"},{"b":"400000","o":"89ED65"},{"b":"400000","o":"AFDD4B"},{"b":"400000","o":"BAC904"},{"b":"7FD76AE84000","o":"7DF3"},{"b":"7FD769E98000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Linux", "release" : "3.10.0-123.9.3.el7.x86_64", "version" : "#1 SMP Thu Nov 6 15:06:03 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF190FE000", "elfType" : 3 }, { "b" : "7FD76AE84000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FD76AC7C000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FD76AA78000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FD76A771000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FD76A46F000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FD76A259000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FD769E98000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FD76B0A0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5e979]
|
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf07861]
|
mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeb491]
|
mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x2D0) [0xd886a0]
|
mongod(_ZN5mongo22WiredTigerRecoveryUnit8_txnOpenEPNS_16OperationContextE+0xF4) [0xd7d134]
|
mongod(_ZN5mongo22WiredTigerRecoveryUnit10getSessionEPNS_16OperationContextE+0x22) [0xd7d192]
|
mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x35) [0xd7d1d5]
|
mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEi+0x55) [0xd6a485]
|
mongod(_ZNK5mongo22BtreeBasedAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xa73f15]
|
mongod(_ZN5mongo11IDHackStage4workEPm+0xD0) [0x9f5710]
|
mongod(_ZN5mongo11UpdateStage4workEPm+0x7C) [0xa2273c]
|
mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbc6464]
|
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbc6814]
|
mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x3D) [0xbc6e7d]
|
mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x118) [0xb123c8]
|
mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbb+0xF48) [0xc33a78]
|
mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0x2EF) [0xc9bebf]
|
mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x65) [0xc9ed65]
|
mongod(_ZN5mongo10threadpool6Worker4loopERKSs+0x2FB) [0xefdd4b]
|
mongod(+0xBAC904) [0xfac904]
|
libpthread.so.0(+0x7DF3) [0x7fd76ae8bdf3]
|
libc.so.6(clone+0x6D) [0x7fd769f8e1ad]
|
----- END BACKTRACE -----
|
2016-06-20T12:43:50.899+0800 I - [repl writer worker 6]
|
|
***aborting after fassert() failure
|
|