-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.2.0
-
Component/s: WiredTiger
-
Labels:None
-
ALL
I have monitoring this issue for a long time. And mongodb had upgraded from 3.0, 3.1, 3.2 it's like all versions meet this issue.
I am using CentOS 7 on a machine Xeon E3 1230V2 CPU and 32GB memory. Only running one mongdb instance. When system clean boot up, the mongodb can work very well for a long time, from weeks to month. But it will finally crash at some point of time. I see the reason in trace log is almost same, which is wiredtiger crash.
There is no help to restart mongodb, it will continue to crash again and again. But when I reboot linux, mongodb will go back to normal, and it can work for another weeks or month. So I am wondering if it is caused by the system buffer. I used free command to trace the memory usage. It's like when the buffer reach to 11GB (32GB total memory), the db will crash at sometime.
I pasted traceback below.
2015-12-23T01:15:48.098+0800 E STORAGE [conn144] WiredTiger (0) [1450804548:98814][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: read checksum error for 12288B block at offset 8978432: calculated block checksum of 368775483 doesn't match expected checksum of 3419546407
2015-12-23T01:15:48.098+0800 E STORAGE [conn144] WiredTiger (0) [1450804548:98899][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: index-5-8308272683202114838.wt: encountered an illegal file format or internal value
2015-12-23T01:15:48.098+0800 E STORAGE [conn144] WiredTiger (-31804) [1450804548:98912][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-12-23T01:15:48.098+0800 I - [conn144] Fatal Assertion 28558
2015-12-23T01:15:48.099+0800 I - [conn144]
***aborting after fassert() failure
2015-12-23T01:15:48.123+0800 F - [conn144] Got signal: 6 (Aborted).
0x12e7b12 0x12e6c79 0x12e7482 0x7ff422bda130 0x7ff42283f5d7 0x7ff422840cc8 0x12724d2 0x10725a3 0x1a2682c 0x1a269cd 0x1a26db4 0x19662d6 0x198285a 0x1988359 0x19a42a5 0x19738d0 0x19bf0ac 0x10541aa 0xc8bb1b 0xbd579a 0xc0a3b4 0xe04b05 0xe051c9 0xe052c5 0xb84f2a 0xb86f25 0xb872ee 0xb8a398 0xba23a9 0xba3046 0xb012e0 0xcab76d 0xcae026 0x99513c 0x12956fd 0x7ff422bd2df5 0x7ff4229001ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EE7B12"},{"b":"400000","o":"EE6C79"},{"b":"400000","o":"EE7482"},{"b":"7FF422BCB000","o":"F130"},{"b":"7FF42280A000","o":"355D7"},{"b":"7FF42280A000","o":"36CC8"},{"b":"400000","o":"E724D2"},{"b":"400000","o":"C725A3"},{"b":"400000","o":"162682C"},{"b":"400000","o":"16269CD"},{"b":"400000","o":"1626DB4"},{"b":"400000","o":"15662D6"},{"b":"400000","o":"158285A"},{"b":"400000","o":"1588359"},{"b":"400000","o":"15A42A5"},{"b":"400000","o":"15738D0"},{"b":"400000","o":"15BF0AC"},{"b":"400000","o":"C541AA"},{"b":"400000","o":"88BB1B"},{"b":"400000","o":"7D579A"},{"b":"400000","o":"80A3B4"},{"b":"400000","o":"A04B05"},{"b":"400000","o":"A051C9"},{"b":"400000","o":"A052C5"},{"b":"400000","o":"784F2A"},{"b":"400000","o":"786F25"},{"b":"400000","o":"7872EE"},{"b":"400000","o":"78A398"},{"b":"400000","o":"7A23A9"},{"b":"400000","o":"7A3046"},{"b":"400000","o":"7012E0"},{"b":"400000","o":"8AB76D"},{"b":"400000","o":"8AE026"},{"b":"400000","o":"59513C"},{"b":"400000","o":"E956FD"},{"b":"7FF422BCB000","o":"7DF5"},{"b":"7FF42280A000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.2.0", "gitVersion" : "45d947729a0315accb6d4f15a6b06be6d9c19fe7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-229.20.1.el7.x86_64", "version" : "#1 SMP Tue Nov 3 19:10:07 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4693068F80266291D0D607EF2ED5A6B255864A0F" }, { "b" : "7FFC6CA7D000", "elfType" : 3, "buildId" : "248B82562C046037E264A35BBF5233A2E4A8FBE3" }, { "b" : "7FF423DF9000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BB96EE99138B19FECDAB55E80A1728B648ECAD50" }, { "b" : "7FF423A12000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "B154203FB7C05AEE29D5D6F6C000305191209FE4" }, { "b" : "7FF42380A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "7376A07360DC57189A8F92B20AA4AA1CAEA80551" }, { "b" : "7FF423606000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "4DFEE4EA9AE8FDD4C71BD4CCC0727222F19DF810" }, { "b" : "7FF4232FF000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "405EACD649720B8668FFBBA197CBF030A7EF6296" }, { "b" : "7FF422FFD000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "A1AA62B29765BE03A36BF927B047EEEF8696EEC6" }, { "b" : "7FF422DE7000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "5D3D7256AE68BCFF41E312A24825ED80ECA88A73" }, { "b" : "7FF422BCB000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "12F30315D4F4A2FE58B1977405C8B5515861E66B" }, { "b" : "7FF42280A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "C31FFE7942BFD77B2FCA8F9BD5709D387A86D3BC" }, { "b" : "7FF424066000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9866E1D2BA61EBB4CE4F009FACDAACC24EF3B804" }, { "b" : "7FF4225BE000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "34672D541C8C9C5C1C25CB4F3F332CC9D3E604AD" }, { "b" : "7FF4222DB000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45CB7F6CD322F5B55FF8B635F7EC1578631CCAEA" }, { "b" : "7FF4220D7000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7FF421EA5000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "23A2D854538903E2B84EF0882046DD95522C8B59" }, { "b" : "7FF421C8F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "E45643F27F3B3E960F3691AFC6EC27A98EF7B46B" }, { "b" : "7FF421A80000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "F4A3D5E7E23F871751CA8F250421F8CF83447AD2" }, { "b" : "7FF42187C000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FF421662000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "AC596E865AF0D14B10F7B707F47D2031AD6D68DC" }, { "b" : "7FF42143D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7FF4211DC000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "298B19C64B19995F2AA4DA7B852E90BA5302F630" }, { "b" : "7FF420FB7000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "218D03D1F6CF1A099A4D467B5E8ECF4F2BF45750" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12e7b12]
mongod(+0xEE6C79) [0x12e6c79]
mongod(+0xEE7482) [0x12e7482]
libpthread.so.0(+0xF130) [0x7ff422bda130]
libc.so.6(gsignal+0x37) [0x7ff42283f5d7]
libc.so.6(abort+0x148) [0x7ff422840cc8]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12724d2]
mongod(+0xC725A3) [0x10725a3]
mongod(__wt_eventv+0x40C) [0x1a2682c]
mongod(__wt_err+0x8D) [0x1a269cd]
mongod(__wt_panic+0x24) [0x1a26db4]
mongod(__wt_bm_read+0x76) [0x19662d6]
mongod(__wt_bt_read+0x1EA) [0x198285a]
mongod(__wt_page_in_func+0x729) [0x1988359]
mongod(__wt_row_search+0x8E5) [0x19a42a5]
mongod(__wt_btcur_search+0xCE0) [0x19738d0]
mongod(+0x15BF0AC) [0x19bf0ac]
mongod(+0xC541AA) [0x10541aa]
mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x3B) [0xc8bb1b]
mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbd579a]
mongod(_ZN5mongo11UpdateStage4workEPm+0x3B4) [0xc0a3b4]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe04b05]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe051c9]
mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe052c5]
mongod(_ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE+0x6FA) [0xb84f2a]
mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x2B5) [0xb86f25]
mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x1DE) [0xb872ee]
mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x248) [0xb8a398]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x3F9) [0xba23a9]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3E6) [0xba3046]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb012e0]
mongod(+0x8AB76D) [0xcab76d]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6C6) [0xcae026]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x99513c]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x12956fd]
libpthread.so.0(+0x7DF5) [0x7ff422bd2df5]
libc.so.6(clone+0x6D) [0x7ff4229001ad]
----- END BACKTRACE -----
Another trace back
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EE7B12"},{"b":"400000","o":"EE6C79"},{"b":"400000","o":"EE7482"},{"b":"7F1F5D21E000","o":"F100"},{"b":"7F1F5CE5D000","o":"355F7"},{"b":"7F1F5CE5D000","o":"36CE8"},{"b":"400000","o":"E724D2"},{"b":"400000","o":"C725A3"},{"b":"400000","o":"162682C"},{"b":"400000","o":"16269CD"},{"b":"400000","o":"1626DB4"},{"b":"400000","o":"15662D6"},{"b":"400000","o":"158285A"},{"b":"400000","o":"1588359"},{"b":"400000","o":"15A42A5"},{"b":"400000","o":"15738D0"},{"b":"400000","o":"15BF0AC"},{"b":"400000","o":"C541AA"},{"b":"400000","o":"88BB1B"},{"b":"400000","o":"7D579A"},{"b":"400000","o":"A04B05"},{"b":"400000","o":"A051C9"},{"b":"400000","o":"723F0D"},{"b":"400000","o":"7A23A9"},{"b":"400000","o":"7A3046"},{"b":"400000","o":"7012E0"},{"b":"400000","o":"8AB76D"},{"b":"400000","o":"8AE026"},{"b":"400000","o":"59513C"},{"b":"400000","o":"E956FD"},{"b":"7F1F5D21E000","o":"7DC5"},{"b":"7F1F5CE5D000","o":"F621D"}],"processInfo":{ "mongodbVersion" : "3.2.0", "gitVersion" : "45d947729a0315accb6d4f15a6b06be6d9c19fe7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.3.1.el7.x86_64", "version" : "#1 SMP Wed Dec 9 14:09:15 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4693068F80266291D0D607EF2ED5A6B255864A0F" }, { "b" : "7FFC81488000", "elfType" : 3, "buildId" : "6D522BD99C31E96B22FABF33083A3E17E0A09D79" }, { "b" : "7F1F5E44D000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7E24873B9C50842F0D3285B7D06F872226D17262" }, { "b" : "7F1F5E066000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "9523142ED92FE40B988E6EED854D79BF587954C5" }, { "b" : "7F1F5DE5E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CB0D2C9F29DBD13C47E7D2EEFB94B35835698CCA" }, { "b" : "7F1F5DC5A000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7F1F5D952000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "8941888BF8EE9CED585599BE5397A385FC1C73CE" }, { "b" : "7F1F5D650000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F9DF294FB70243549DCB643F1322BB20E70E9FE8" }, { "b" : "7F1F5D43A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7F1F5D21E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "723F0AC75EF88E778940AE8A8BC30141D85B116A" }, { "b" : "7F1F5CE5D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "2434E3F90543A7D58D480E8B8C4FAFACEA7FF1D6" }, { "b" : "7F1F5E6BA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "09E1BB4D034C7263810A41100647068858A7ECB6" }, { "b" : "7F1F5CC11000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7F1F5C92C000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7F1F5C728000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7F1F5C4F6000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7F1F5C2E0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7F1F5C0D1000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7F1F5BECD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1F5BCB3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7F1F5BA8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F1F5B82D000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7F1F5B608000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12e7b12]
mongod(+0xEE6C79) [0x12e6c79]
mongod(+0xEE7482) [0x12e7482]
libpthread.so.0(+0xF100) [0x7f1f5d22d100]
libc.so.6(gsignal+0x37) [0x7f1f5ce925f7]
libc.so.6(abort+0x148) [0x7f1f5ce93ce8]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12724d2]
mongod(+0xC725A3) [0x10725a3]
mongod(__wt_eventv+0x40C) [0x1a2682c]
mongod(__wt_err+0x8D) [0x1a269cd]
mongod(__wt_panic+0x24) [0x1a26db4]
mongod(__wt_bm_read+0x76) [0x19662d6]
mongod(__wt_bt_read+0x1EA) [0x198285a]
mongod(__wt_page_in_func+0x729) [0x1988359]
mongod(__wt_row_search+0x8E5) [0x19a42a5]
mongod(__wt_btcur_search+0xCE0) [0x19738d0]
mongod(+0x15BF0AC) [0x19bf0ac]
mongod(+0xC541AA) [0x10541aa]
mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x3B) [0xc8bb1b]
mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbd579a]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe04b05]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe051c9]
mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x98D) [0xb23f0d]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x3F9) [0xba23a9]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3E6) [0xba3046]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb012e0]
mongod(+0x8AB76D) [0xcab76d]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6C6) [0xcae026]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x99513c]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x12956fd]
libpthread.so.0(+0x7DC5) [0x7f1f5d225dc5]
libc.so.6(clone+0x6D) [0x7f1f5cf5321d]
----- END BACKTRACE -----