[SERVER-24659] The mongodb is crashed Created: 20/Jun/16  Updated: 14/Jul/16  Resolved: 20/Jun/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: waynot Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

My mongodb which version is 3.0.7 is crashed, the following is detailed information:

2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1466397830:886662][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: read checksum error for 16384B block at offset 7593738240: calculated block checksum of 384598265 doesn't match expected checksum of 2923502613
2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1466397830:886748][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: collection-143-1451283047013864403.wt: encountered an illegal file format or internal value
2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (-31804) [1466397830:886769][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 12] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 7] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 10] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 4] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 1] Fatal Assertion 28558
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 6] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 5] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 0] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 8] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 13] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 15] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 9] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 14] Fatal Assertion 28559
2016-06-20T12:43:50.886+0800 I -        [repl writer worker 3] Fatal Assertion 28559
2016-06-20T12:43:50.887+0800 I -        [repl writer worker 11] Fatal Assertion 28559
2016-06-20T12:43:50.887+0800 I -        [repl writer worker 2] Fatal Assertion 28559
2016-06-20T12:43:50.890+0800 I ACCESS   [conn108] Successfully authenticated as principal qxbdev on iEnterprise
2016-06-20T12:43:50.891+0800 I NETWORK  [initandlisten] connection accepted from 10.47.90.78:54524 #110 (56 connections now open)
2016-06-20T12:43:50.896+0800 I ACCESS   [conn107] Successfully authenticated as principal qxbdev on iEnterprise
2016-06-20T12:43:50.898+0800 I NETWORK  [conn107] end connection 10.47.90.78:54520 (55 connections now open)
2016-06-20T12:43:50.898+0800 I NETWORK  [initandlisten] connection accepted from 10.47.90.78:54525 #111 (56 connections now open)
2016-06-20T12:43:50.898+0800 I CONTROL  [repl writer worker 6] 
 0xf5e979 0xf07861 0xeeb491 0xd886a0 0xd7d134 0xd7d192 0xd7d1d5 0xd6a485 0xa73f15 0x9f5710 0xa2273c 0xbc6464 0xbc6814 0xbc6e7d 0xb123c8 0xc33a78 0xc9bebf 0xc9ed65 0xefdd4b 0xfac904 0x7fd76ae8bdf3 0x7fd769f8e1ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5E979"},{"b":"400000","o":"B07861"},{"b":"400000","o":"AEB491"},{"b":"400000","o":"9886A0"},{"b":"400000","o":"97D134"},{"b":"400000","o":"97D192"},{"b":"400000","o":"97D1D5"},{"b":"400000","o":"96A485"},{"b":"400000","o":"673F15"},{"b":"400000","o":"5F5710"},{"b":"400000","o":"62273C"},{"b":"400000","o":"7C6464"},{"b":"400000","o":"7C6814"},{"b":"400000","o":"7C6E7D"},{"b":"400000","o":"7123C8"},{"b":"400000","o":"833A78"},{"b":"400000","o":"89BEBF"},{"b":"400000","o":"89ED65"},{"b":"400000","o":"AFDD4B"},{"b":"400000","o":"BAC904"},{"b":"7FD76AE84000","o":"7DF3"},{"b":"7FD769E98000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Linux", "release" : "3.10.0-123.9.3.el7.x86_64", "version" : "#1 SMP Thu Nov 6 15:06:03 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF190FE000", "elfType" : 3 }, { "b" : "7FD76AE84000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FD76AC7C000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FD76AA78000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FD76A771000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FD76A46F000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FD76A259000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FD769E98000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FD76B0A0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5e979]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf07861]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeb491]
 mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x2D0) [0xd886a0]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit8_txnOpenEPNS_16OperationContextE+0xF4) [0xd7d134]
 mongod(_ZN5mongo22WiredTigerRecoveryUnit10getSessionEPNS_16OperationContextE+0x22) [0xd7d192]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x35) [0xd7d1d5]
 mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEi+0x55) [0xd6a485]
 mongod(_ZNK5mongo22BtreeBasedAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xa73f15]
 mongod(_ZN5mongo11IDHackStage4workEPm+0xD0) [0x9f5710]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x7C) [0xa2273c]
 mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbc6464]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbc6814]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x3D) [0xbc6e7d]
 mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x118) [0xb123c8]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbb+0xF48) [0xc33a78]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0x2EF) [0xc9bebf]
 mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x65) [0xc9ed65]
 mongod(_ZN5mongo10threadpool6Worker4loopERKSs+0x2FB) [0xefdd4b]
 mongod(+0xBAC904) [0xfac904]
 libpthread.so.0(+0x7DF3) [0x7fd76ae8bdf3]
 libc.so.6(clone+0x6D) [0x7fd769f8e1ad]
-----  END BACKTRACE  -----
2016-06-20T12:43:50.899+0800 I -        [repl writer worker 6]
 
***aborting after fassert() failure



 Comments   
Comment by Ramon Fernandez Marina [ 20/Jun/16 ]

jin zhihao, unfortunately the following lines indicate that mongod found corrupted data:

2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1466397830:886662][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: read checksum error for 16384B block at offset 7593738240: calculated block checksum of 384598265 doesn't match expected checksum of 2923502613
2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1466397830:886748][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: collection-143-1451283047013864403.wt: encountered an illegal file format or internal value
2016-06-20T12:43:50.886+0800 E STORAGE  [repl writer worker 1] WiredTiger (-31804) [1466397830:886769][12706:0x7fd756855700], file:collection-143-1451283047013864403.wt, cursor.search: the process must exit and restart: WT_PANIC: WiredTiger library panic

The simplest approach is to resync the data from a healthy node. Alternatively you may wan to run repair. I'd also make sure you always shut down your database cleanly and that your storage layer is healthy, as flaky storage layers are often to blame for corrupted files.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 04:07:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.