[SERVER-25923] Server stop on after invarient failure Created: 01/Sep/16  Updated: 13/Dec/16  Resolved: 13/Dec/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Denis Laboureyras Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello,

I've got 3 servers on last stable version of MongoDB (3.2.9), with replication (without sharding)
My servers are crashing really often these days, and I don't know really why.

I've managed to catch some backtrace in the logs, but they are not really helpful to me... I've pasted them at the end of the message.
Can you help me fix that bug ? Thanks a lot for your answer.

Kind regards,
Denis

----- BEGIN BACKTRACE -----
{"backtrace":[{
"b":"400000","o":"1121902","s":"_ZN5mongo15printStackTraceERSo"},
{"b":"400000","o":"10BEC18","s":"_ZN5mongo10logContextEPKc"},
{"b":"400000","o":"10A5773","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},
{"b":"400000","o":"E41F6F","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},
{"b":"400000","o":"E402B3","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},
{"b":"400000","o":"E21637","s":"_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},
{"b":"400000","o":"97037A","s":"_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl"},
{"b":"400000","o":"73FCB2","s":"_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},
{"b":"400000","o":"73FF34","s":"_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},
{"b":"400000","o":"7403D7","s":"_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE"},
{"b":"400000","o":"71CF15","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b"},
{"b":"400000","o":"71D1B1","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb"},
{"b":"400000","o":"71D38D","s":"_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb"},
{"b":"400000","o":"830A21","s":"_ZN5mongo18WriteBatchExecutor10insertManyEPNS0_16ExecInsertsStateEmmPNS_5CurOpEPSt6vectorIPNS_16WriteErrorDetailESaIS7_EEb"},
{"b":"400000","o":"83221B","s":"_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE"},
{"b":"400000","o":"834CE3","s":"_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE"},
{"b":"400000","o":"8352CB","s":"_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE"},
{"b":"400000","o":"8384D0","s":"_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},
{"b":"400000","o":"85B006","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},
{"b":"400000","o":"85C2F5","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},
{"b":"400000","o":"76ED10","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},
{"b":"400000","o":"99236A"},{"b":"400000","o":"995926","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},
{"b":"400000","o":"5C3F6A","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},
{"b":"400000","o":"10CD111","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},
{"b":"76A57B51E000","o":"76AA"},{"b":"76A57B154000","o":"10713D","s":"clone"}]
,"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", 
"compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.14.32-xxxx-grs-ipv6-64", "version" : "#6 SMP Wed Jan 20 17:52:44 CET 2016", "machine" : "x86_64" }, 
"somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "1F47133C759EFF79F90A6EBF640DC97B744D70C1" }, { "b" : "76A57D236000", "elfType" : 3, "buildId" : "FAF400EE48C6DC7D3D021FC95AA21E92ED9541BC" }, { "b" : "76A57C4AB000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "96D927A52B6A405C147AC4D3F8A6F14CC31316BA" }, { "b" : "76A57C067000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "039AD0290D6DDCD62FFAAFF6D241FD313938E654" }, { "b" : "76A57BE5F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "0370F7BC9F3A530FBB3D7918E67713E9BFF68FD8" }, { "b" : "76A57BC5B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "7B3B05F668FF51BFFDF2B2B560934813C083A948" }, { "b" : "76A57B953000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "10EED81FF44190C88FCD4D807248BE110352D5FC" }, { "b" : "76A57B73C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "0C3C07EE15CFA81346847A679E8444B876D9CC58" }, { "b" : "76A57B51E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "41D72FB9BBC5E6FCE5654DC0CF23BC614782B0DA" }, { "b" : "76A57B154000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5CCD94C4E3483DF05BE240FF1FB8A3F53794CC6F" }, { "b" : "76A57C714000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "AFE4833057694750DE5F6F5D713F7CB6CC4F195A" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1521902]
mongod(_ZN5mongo10logContextEPKc+0x168) [0x14bec18]
mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xF3) [0x14a5773]
mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0xEF) [0x1241f6f]
mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x53) [0x12402b3]
mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0x97) [0x1221637]
mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x1DA) [0xd7037a]
mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0xB2) [0xb3fcb2]
mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x1C4) [0xb3ff34]
mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE+0x77) [0xb403d7]
mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x5A5) [0xb1cf15]
mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x241) [0xb1d1b1]
mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb+0x6D) [0xb1d38d]
mongod(_ZN5mongo18WriteBatchExecutor10insertManyEPNS0_16ExecInsertsStateEmmPNS_5CurOpEPSt6vectorIPNS_16WriteErrorDetailESaIS7_EEb+0xC81) [0xc30a21]
mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x67B) [0xc3221b]
mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x63) [0xc34ce3]
mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x1DB) [0xc352cb]
mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x280) [0xc384d0]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x676) [0xc5b006]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x895) [0xc5c2f5]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x260) [0xb6ed10]
mongod(+0x99236A) [0xd9236a]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7D6) [0xd95926]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEA) [0x9c3f6a]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x311) [0x14cd111]
libpthread.so.0(+0x76AA) [0x76a57b5256aa]
libc.so.6(clone+0x6D) [0x76a57b25b13d]
-----  END BACKTRACE  -----
2016-09-01T14:17:45.104+0200 I -        [conn286]
aborting after invariant() failure



 Comments   
Comment by Kelsey Schubert [ 13/Dec/16 ]

Hi denislaboureyras,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Thomas

Comment by Kelsey Schubert [ 16/Sep/16 ]

Hi denislaboureyras,

We still the logs that Ramon requested to continue to investigate. Would you please let us know when you have uploaded them to the portal?

Thank you,
Thomas

Comment by Ramon Fernandez Marina [ 02/Sep/16 ]

For each affected node, a section of the logs that goes from the last server restart until the invariant failure. When compressed it should not be too big, but I've created a private upload portal that allows uploading big files if needed (JIRA has a limit of 150MB). I think that should provide enough information to get the investigation going.

Thanks,
Ramón.

Comment by Denis Laboureyras [ 02/Sep/16 ]

What do you want exactly ? 1 day of log ?
I've got 8go of logs but I don't think you want all of it

Comment by Ramon Fernandez Marina [ 01/Sep/16 ]

Sorry to hear you're running into issues denislaboureyras. Can you please upload the full logs for the affected nodes to this ticket?

Thanks,
Ramón.

Generated at Thu Feb 08 04:10:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.