[SERVER-22211] WiredTiger crash when Linux System buffer reach to some level Created: 17/Jan/16  Updated: 11/Mar/16  Resolved: 22/Jan/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jerry Zhu Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I have monitoring this issue for a long time. And mongodb had upgraded from 3.0, 3.1, 3.2 it's like all versions meet this issue.

I am using CentOS 7 on a machine Xeon E3 1230V2 CPU and 32GB memory. Only running one mongdb instance. When system clean boot up, the mongodb can work very well for a long time, from weeks to month. But it will finally crash at some point of time. I see the reason in trace log is almost same, which is wiredtiger crash.

There is no help to restart mongodb, it will continue to crash again and again. But when I reboot linux, mongodb will go back to normal, and it can work for another weeks or month. So I am wondering if it is caused by the system buffer. I used free command to trace the memory usage. It's like when the buffer reach to 11GB (32GB total memory), the db will crash at sometime.

I pasted traceback below.

2015-12-23T01:15:48.098+0800 E STORAGE  [conn144] WiredTiger (0) [1450804548:98814][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: read checksum error for 12288B block at offset 8978432: calculated block checksum of 368775483 doesn't match expected checksum of 3419546407
2015-12-23T01:15:48.098+0800 E STORAGE  [conn144] WiredTiger (0) [1450804548:98899][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: index-5-8308272683202114838.wt: encountered an illegal file format or internal value
2015-12-23T01:15:48.098+0800 E STORAGE  [conn144] WiredTiger (-31804) [1450804548:98912][1289:0x7ff411654700], file:index-5-8308272683202114838.wt, WT_CURSOR.search: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-12-23T01:15:48.098+0800 I -        [conn144] Fatal Assertion 28558
2015-12-23T01:15:48.099+0800 I -        [conn144]
 
***aborting after fassert() failure
 
 
2015-12-23T01:15:48.123+0800 F -        [conn144] Got signal: 6 (Aborted).
 
 0x12e7b12 0x12e6c79 0x12e7482 0x7ff422bda130 0x7ff42283f5d7 0x7ff422840cc8 0x12724d2 0x10725a3 0x1a2682c 0x1a269cd 0x1a26db4 0x19662d6 0x198285a 0x1988359 0x19a42a5 0x19738d0 0x19bf0ac 0x10541aa 0xc8bb1b 0xbd579a 0xc0a3b4 0xe04b05 0xe051c9 0xe052c5 0xb84f2a 0xb86f25 0xb872ee 0xb8a398 0xba23a9 0xba3046 0xb012e0 0xcab76d 0xcae026 0x99513c 0x12956fd 0x7ff422bd2df5 0x7ff4229001ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EE7B12"},{"b":"400000","o":"EE6C79"},{"b":"400000","o":"EE7482"},{"b":"7FF422BCB000","o":"F130"},{"b":"7FF42280A000","o":"355D7"},{"b":"7FF42280A000","o":"36CC8"},{"b":"400000","o":"E724D2"},{"b":"400000","o":"C725A3"},{"b":"400000","o":"162682C"},{"b":"400000","o":"16269CD"},{"b":"400000","o":"1626DB4"},{"b":"400000","o":"15662D6"},{"b":"400000","o":"158285A"},{"b":"400000","o":"1588359"},{"b":"400000","o":"15A42A5"},{"b":"400000","o":"15738D0"},{"b":"400000","o":"15BF0AC"},{"b":"400000","o":"C541AA"},{"b":"400000","o":"88BB1B"},{"b":"400000","o":"7D579A"},{"b":"400000","o":"80A3B4"},{"b":"400000","o":"A04B05"},{"b":"400000","o":"A051C9"},{"b":"400000","o":"A052C5"},{"b":"400000","o":"784F2A"},{"b":"400000","o":"786F25"},{"b":"400000","o":"7872EE"},{"b":"400000","o":"78A398"},{"b":"400000","o":"7A23A9"},{"b":"400000","o":"7A3046"},{"b":"400000","o":"7012E0"},{"b":"400000","o":"8AB76D"},{"b":"400000","o":"8AE026"},{"b":"400000","o":"59513C"},{"b":"400000","o":"E956FD"},{"b":"7FF422BCB000","o":"7DF5"},{"b":"7FF42280A000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.2.0", "gitVersion" : "45d947729a0315accb6d4f15a6b06be6d9c19fe7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-229.20.1.el7.x86_64", "version" : "#1 SMP Tue Nov 3 19:10:07 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4693068F80266291D0D607EF2ED5A6B255864A0F" }, { "b" : "7FFC6CA7D000", "elfType" : 3, "buildId" : "248B82562C046037E264A35BBF5233A2E4A8FBE3" }, { "b" : "7FF423DF9000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BB96EE99138B19FECDAB55E80A1728B648ECAD50" }, { "b" : "7FF423A12000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "B154203FB7C05AEE29D5D6F6C000305191209FE4" }, { "b" : "7FF42380A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "7376A07360DC57189A8F92B20AA4AA1CAEA80551" }, { "b" : "7FF423606000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "4DFEE4EA9AE8FDD4C71BD4CCC0727222F19DF810" }, { "b" : "7FF4232FF000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "405EACD649720B8668FFBBA197CBF030A7EF6296" }, { "b" : "7FF422FFD000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "A1AA62B29765BE03A36BF927B047EEEF8696EEC6" }, { "b" : "7FF422DE7000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "5D3D7256AE68BCFF41E312A24825ED80ECA88A73" }, { "b" : "7FF422BCB000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "12F30315D4F4A2FE58B1977405C8B5515861E66B" }, { "b" : "7FF42280A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "C31FFE7942BFD77B2FCA8F9BD5709D387A86D3BC" }, { "b" : "7FF424066000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9866E1D2BA61EBB4CE4F009FACDAACC24EF3B804" }, { "b" : "7FF4225BE000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "34672D541C8C9C5C1C25CB4F3F332CC9D3E604AD" }, { "b" : "7FF4222DB000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45CB7F6CD322F5B55FF8B635F7EC1578631CCAEA" }, { "b" : "7FF4220D7000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7FF421EA5000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "23A2D854538903E2B84EF0882046DD95522C8B59" }, { "b" : "7FF421C8F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "E45643F27F3B3E960F3691AFC6EC27A98EF7B46B" }, { "b" : "7FF421A80000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "F4A3D5E7E23F871751CA8F250421F8CF83447AD2" }, { "b" : "7FF42187C000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FF421662000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "AC596E865AF0D14B10F7B707F47D2031AD6D68DC" }, { "b" : "7FF42143D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7FF4211DC000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "298B19C64B19995F2AA4DA7B852E90BA5302F630" }, { "b" : "7FF420FB7000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "218D03D1F6CF1A099A4D467B5E8ECF4F2BF45750" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12e7b12]
 mongod(+0xEE6C79) [0x12e6c79]
 mongod(+0xEE7482) [0x12e7482]
 libpthread.so.0(+0xF130) [0x7ff422bda130]
 libc.so.6(gsignal+0x37) [0x7ff42283f5d7]
 libc.so.6(abort+0x148) [0x7ff422840cc8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12724d2]
 mongod(+0xC725A3) [0x10725a3]
 mongod(__wt_eventv+0x40C) [0x1a2682c]
 mongod(__wt_err+0x8D) [0x1a269cd]
 mongod(__wt_panic+0x24) [0x1a26db4]
 mongod(__wt_bm_read+0x76) [0x19662d6]
 mongod(__wt_bt_read+0x1EA) [0x198285a]
 mongod(__wt_page_in_func+0x729) [0x1988359]
 mongod(__wt_row_search+0x8E5) [0x19a42a5]
 mongod(__wt_btcur_search+0xCE0) [0x19738d0]
 mongod(+0x15BF0AC) [0x19bf0ac]
 mongod(+0xC541AA) [0x10541aa]
 mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x3B) [0xc8bb1b]
 mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbd579a]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x3B4) [0xc0a3b4]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe04b05]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe051c9]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe052c5]
 mongod(_ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE+0x6FA) [0xb84f2a]
 mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x2B5) [0xb86f25]
 mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x1DE) [0xb872ee]
 mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x248) [0xb8a398]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x3F9) [0xba23a9]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3E6) [0xba3046]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb012e0]
 mongod(+0x8AB76D) [0xcab76d]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6C6) [0xcae026]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x99513c]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x12956fd]
 libpthread.so.0(+0x7DF5) [0x7ff422bd2df5]
 libc.so.6(clone+0x6D) [0x7ff4229001ad]
-----  END BACKTRACE  -----

Another trace back

----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EE7B12"},{"b":"400000","o":"EE6C79"},{"b":"400000","o":"EE7482"},{"b":"7F1F5D21E000","o":"F100"},{"b":"7F1F5CE5D000","o":"355F7"},{"b":"7F1F5CE5D000","o":"36CE8"},{"b":"400000","o":"E724D2"},{"b":"400000","o":"C725A3"},{"b":"400000","o":"162682C"},{"b":"400000","o":"16269CD"},{"b":"400000","o":"1626DB4"},{"b":"400000","o":"15662D6"},{"b":"400000","o":"158285A"},{"b":"400000","o":"1588359"},{"b":"400000","o":"15A42A5"},{"b":"400000","o":"15738D0"},{"b":"400000","o":"15BF0AC"},{"b":"400000","o":"C541AA"},{"b":"400000","o":"88BB1B"},{"b":"400000","o":"7D579A"},{"b":"400000","o":"A04B05"},{"b":"400000","o":"A051C9"},{"b":"400000","o":"723F0D"},{"b":"400000","o":"7A23A9"},{"b":"400000","o":"7A3046"},{"b":"400000","o":"7012E0"},{"b":"400000","o":"8AB76D"},{"b":"400000","o":"8AE026"},{"b":"400000","o":"59513C"},{"b":"400000","o":"E956FD"},{"b":"7F1F5D21E000","o":"7DC5"},{"b":"7F1F5CE5D000","o":"F621D"}],"processInfo":{ "mongodbVersion" : "3.2.0", "gitVersion" : "45d947729a0315accb6d4f15a6b06be6d9c19fe7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.3.1.el7.x86_64", "version" : "#1 SMP Wed Dec 9 14:09:15 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4693068F80266291D0D607EF2ED5A6B255864A0F" }, { "b" : "7FFC81488000", "elfType" : 3, "buildId" : "6D522BD99C31E96B22FABF33083A3E17E0A09D79" }, { "b" : "7F1F5E44D000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7E24873B9C50842F0D3285B7D06F872226D17262" }, { "b" : "7F1F5E066000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "9523142ED92FE40B988E6EED854D79BF587954C5" }, { "b" : "7F1F5DE5E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CB0D2C9F29DBD13C47E7D2EEFB94B35835698CCA" }, { "b" : "7F1F5DC5A000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7F1F5D952000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "8941888BF8EE9CED585599BE5397A385FC1C73CE" }, { "b" : "7F1F5D650000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F9DF294FB70243549DCB643F1322BB20E70E9FE8" }, { "b" : "7F1F5D43A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7F1F5D21E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "723F0AC75EF88E778940AE8A8BC30141D85B116A" }, { "b" : "7F1F5CE5D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "2434E3F90543A7D58D480E8B8C4FAFACEA7FF1D6" }, { "b" : "7F1F5E6BA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "09E1BB4D034C7263810A41100647068858A7ECB6" }, { "b" : "7F1F5CC11000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7F1F5C92C000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7F1F5C728000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7F1F5C4F6000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7F1F5C2E0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7F1F5C0D1000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7F1F5BECD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1F5BCB3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7F1F5BA8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F1F5B82D000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7F1F5B608000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12e7b12]
 mongod(+0xEE6C79) [0x12e6c79]
 mongod(+0xEE7482) [0x12e7482]
 libpthread.so.0(+0xF100) [0x7f1f5d22d100]
 libc.so.6(gsignal+0x37) [0x7f1f5ce925f7]
 libc.so.6(abort+0x148) [0x7f1f5ce93ce8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12724d2]
 mongod(+0xC725A3) [0x10725a3]
 mongod(__wt_eventv+0x40C) [0x1a2682c]
 mongod(__wt_err+0x8D) [0x1a269cd]
 mongod(__wt_panic+0x24) [0x1a26db4]
 mongod(__wt_bm_read+0x76) [0x19662d6]
 mongod(__wt_bt_read+0x1EA) [0x198285a]
 mongod(__wt_page_in_func+0x729) [0x1988359]
 mongod(__wt_row_search+0x8E5) [0x19a42a5]
 mongod(__wt_btcur_search+0xCE0) [0x19738d0]
 mongod(+0x15BF0AC) [0x19bf0ac]
 mongod(+0xC541AA) [0x10541aa]
 mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x3B) [0xc8bb1b]
 mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbd579a]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe04b05]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe051c9]
 mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x98D) [0xb23f0d]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x3F9) [0xba23a9]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3E6) [0xba3046]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb012e0]
 mongod(+0x8AB76D) [0xcab76d]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6C6) [0xcae026]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x99513c]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x12956fd]
 libpthread.so.0(+0x7DC5) [0x7f1f5d225dc5]
 libc.so.6(clone+0x6D) [0x7f1f5cf5321d]
-----  END BACKTRACE  -----



 Comments   
Comment by Jerry Zhu [ 11/Mar/16 ]

Hi Thomas,

After about one month monitoring, I think I find the root cause. If the linux server never sleep, the mongo server is very stable and will not crash like before. If I sleep the linux server and wake up, it will crash like this jira reported. Though, the database server should not sleep. I comment here as a very special case.

Comment by Kelsey Schubert [ 22/Jan/16 ]

Hi napoleonn,

The logs you have posted indicate either defective memory or a kernel bug.

I would recommend running diagnostic tests on your memory.

If the memory is not at fault, we recommend disabling hugepages since we have seen some reports that this issue is more likely to occur with it enabled. This setting can be found at /proc/sys/vm/nr_hugepages.

For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag.

Thank you,
Thomas

Comment by Jerry Zhu [ 18/Jan/16 ]

I checked the syslog and paste the error here

Jan 11 21:18:49 matrix4 kernel: BUG: Bad page state in process mongod pfn:41017a
Jan 11 21:18:49 matrix4 kernel: page:ffffea0010405e80 count:1073741824 mapcount:0 mapping: (null) index:0x0
Jan 11 21:18:49 matrix4 kernel: page flags: 0x2fffff00008000(tail)
Jan 11 21:18:49 matrix4 kernel: page dumped because: nonzero _count
Jan 11 21:18:49 matrix4 kernel: Modules linked in: it87 hwmon_vid vfat fat intel_powerclamp coretemp intel_rapl kvm_intel kvm ppdev crc32_pclmul ghash_clmulni_intel aesni_intel tpm_infineon i2c_i801 parport_pc parport iTCO_wdt iTCO_vendor_support mei_me mei sg ie31200_edac video edac_core lrw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep lpc_ich shpchp snd_seq snd_seq_device mfd_core snd_pcm snd_timer snd soundcore gf128mul pcspkr glue_helper ablk_helper cryptd ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic radeon i2c_algo_bit drm_kms_helper crct10dif_pclmul ahci ttm crct10dif_common libahci crc32c_intel drm libata serio_raw r8169 mii i2c_core dm_mirror dm_region_hash dm_log dm_mod
Jan 11 21:18:49 matrix4 kernel: CPU: 4 PID: 7578 Comm: mongod Not tainted 3.10.0-327.3.1.el7.x86_64 #1
Jan 11 21:18:49 matrix4 kernel: Hardware name: Gigabyte Technology Co., Ltd. B85M-D3H/B85M-D3H, BIOS F14 09/19/2014
Jan 11 21:18:49 matrix4 kernel: ffffea0010405e80 000000004228f702 ffff880274d3ba50 ffffffff8163516c
Jan 11 21:18:49 matrix4 kernel: ffff880274d3ba78 ffffffff81630385 ffffea0010405e80 0000000000000000
Jan 11 21:18:49 matrix4 kernel: 000fffff00000000 ffff880274d3bac0 ffffffff811710ad fff00000fe000000
Jan 11 21:18:49 matrix4 kernel: Call Trace:
Jan 11 21:18:49 matrix4 kernel: [<ffffffff8163516c>] dump_stack+0x19/0x1b
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81630385>] bad_page.part.59+0xdf/0xfc
Jan 11 21:18:49 matrix4 kernel: [<ffffffff811710ad>] free_pages_prepare+0x16d/0x190
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81171489>] free_compound_page+0x29/0x40
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81176cdf>] __put_compound_page+0x1f/0x30
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81176d51>] put_compound_page+0x31/0x170
Jan 11 21:18:49 matrix4 kernel: [<ffffffff811772c3>] release_pages+0x203/0x210
Jan 11 21:18:49 matrix4 kernel: [<ffffffff811a971d>] free_pages_and_swap_cache+0xad/0xd0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff8119259c>] tlb_flush_mmu.part.54+0x6c/0xc0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
Jan 11 21:18:49 matrix4 kernel: [<ffffffff8119e85b>] exit_mmap+0xdb/0x1a0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81128af2>] ? __delayacct_add_tsk+0x172/0x1a0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff810782b7>] mmput+0x67/0xf0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff810815ac>] do_exit+0x28c/0xa60
Jan 11 21:18:49 matrix4 kernel: [<ffffffff8108f593>] ? __sigqueue_free.part.11+0x33/0x40
Jan 11 21:18:49 matrix4 kernel: [<ffffffff8108fc7c>] ? __dequeue_signal+0x13c/0x220
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81081dff>] do_group_exit+0x3f/0xa0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81092c10>] get_signal_to_deliver+0x1d0/0x6d0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff810e25c0>] ? futex_wake+0x80/0x160
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81014417>] do_signal+0x57/0x6c0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81014adf>] do_notify_resume+0x5f/0xb0
Jan 11 21:18:49 matrix4 kernel: [<ffffffff81645b7d>] int_signal+0x12/0x17
Jan 11 21:18:49 matrix4 kernel: Disabling lock debugging due to kernel taint

Comment by Daniel Pasette (Inactive) [ 17/Jan/16 ]

That error says you have a checksum mismatch. Can you check you syslog for any storage or memory issues?

Generated at Thu Feb 08 03:59:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.