[SERVER-8547] Mongo crash Created: 13/Feb/13  Updated: 10/Dec/14  Resolved: 28/Oct/13

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ipsy Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC2 XL Linux instance


Attachments: Text File ipsyMongoCrash.log    
Operating System: Linux
Participants:

 Description   

We've experienced several crashes of our production mongod. The server is under reasonable load, with queries backed by indices and no long running queries. Here's one of the crashes:
Sat Feb 9 14:47:39 Backtrace:
0xaffd31 0x558bb9 0x559142 0x7fbf0f8f6500 0x7fbf0ecea60e 0x5c3be8 0x8f480b 0x901a0b 0x908769 0x909072 0x9097c7 0x8dda17 0x8de446 0x8e751a 0x8e8a78 0x8e8c3b 0x82bde4 0x6f1072 0x6e8d71 0x6ea700
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
/lib64/libpthread.so.0(+0xf500) [0x7fbf0f8f6500]
/lib64/libc.so.6(+0x13d60e) [0x7fbf0ecea60e]
/usr/bin/mongod(ZN5mongo20compareElementValuesERKNS_11BSONElementES2+0x778) [0x5c3be8]
/usr/bin/mongod(_ZN5mongo10FieldRange9intersectERKS0_b+0x12b) [0x8f480b]
/usr/bin/mongod(_ZN5mongo13FieldRangeSet19intersectMatchFieldEPKcRKNS_11BSONElementEbb+0x5b) [0x901a0b]
/usr/bin/mongod(_ZN5mongo13FieldRangeSet16handleMatchFieldERKNS_11BSONElementEb+0x1b9) [0x908769]
/usr/bin/mongod(_ZN5mongo13FieldRangeSet4initEb+0xa2) [0x909072]
/usr/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x147) [0x9097c7]
/usr/bin/mongod(ZN5mongo16MultiPlanScanner4initERKNS_7BSONObjES3_S3+0x87) [0x8dda17]
/usr/bin/mongod(ZN5mongo16MultiPlanScanner4makeEPKcRKNS_7BSONObjES5_RKN5boost10shared_ptrIKNS_11ParsedQueryEEES5_NS_18QueryPlanGenerator18RecordedPlanPolicyES5_S5+0x76) [0x8de446]
/usr/bin/mongod(_ZN5mongo15CursorGenerator19setMultiPlanScannerEv+0xea) [0x8e751a]
/usr/bin/mongod(_ZN5mongo15CursorGenerator8generateEv+0x98) [0x8e8a78]
/usr/bin/mongod(_ZN5mongo25NamespaceDetailsTransient9getCursorEPKcRKNS_7BSONObjES5_RKNS_24QueryPlanSelectionPolicyEPbRKN5boost10shared_ptrIKNS_11ParsedQueryEEEbPNS_16QueryPlanSummaryE+0x4b) [0x8e8c3b]
/usr/bin/mongod(_ZN5mongo8runCountEPKcRKNS_7BSONObjERSsRi+0x494) [0x82bde4]
/usr/bin/mongod(_ZN5mongo8CmdCount3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x62) [0x6f1072]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8d71]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xe70) [0x6ea700]

In case it's relevant, we have 3 replication instances: a primary, a secondary, and an arbiter, all within the same ec2 zone.



 Comments   
Comment by Stennie Steneker (Inactive) [ 28/Oct/13 ]

Hi Ipsy,

FYI, objcheck functionality is now enabled by default as of MongoDB 2.4.

I'm closing this issue as there is nothing further for us to investigate at this time.

If you are still seeing this issue (particularly with a newer version of MongoDB, such as 2.4.x) please feel free to open a new issue or comment on this one with the relevant details including logs as well as server and driver versions.

Thanks,
Stephen

Comment by Aaron Staple [ 27/Feb/13 ]

I can't give you precise numbers, but objcheck will do a sanity check of bson sent to mongo, including bson contained in queries and documents to be inserted. This may affect operation latency and server cpu usage. One possibility is that only a fraction of bad bson causes crashes so there's a chance you might not have to wait weeks to see a failure reported. Would it be possible to try out objcheck and see if you can live with the performance?

Comment by Ipsy [ 27/Feb/13 ]

We've only had a few of these crashes, weeks apart. So it would have to stay in production for a while before we could reproduce. What is the expected performance impact of --objcheck?

Thanks.

Comment by Aaron Staple [ 27/Feb/13 ]

Hi Ipsy,

How frequently are you seeing these failures? To help us diagnose the issue, would it be possible for you to run mongod with the --objcheck option and tell us if you see messages such as the following in the log:

Wed Feb 27 10:46:50 [conn1] Assertion: 10307:Client Error: bad object in message

Comment by Ipsy [ 25/Feb/13 ]
  • I just attached a longer log snippet.
  • We're using org.mongodb:mongo-java-driver:2.10.1
  • Not sure which query caused the crash, but the attached log may be of help.

Thanks!

Comment by Ipsy [ 25/Feb/13 ]

mongo log snippet

Comment by Aaron Staple [ 25/Feb/13 ]

Hi Ipsy,

To help us diagnose the issue:

  • Would it be possible to send more of your mongo log?
  • What driver are you using to communicate with mongo?
  • Is there any way you could send the query that triggered this failure?

Thanks

Generated at Thu Feb 08 03:17:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.