[SERVER-2203] Mongodb keep crashing/segfaulting Created: 10/Dec/10  Updated: 12/Jul/16  Resolved: 06/Jan/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.6.5
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: Matthew Ford Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC2 large instance (ubuntu)


Attachments: Text File crash-log-13-12-2010.txt     Text File mongodb-log.txt    
Operating System: Linux
Participants:

 Description   

For the last 2-3 days I have had recurring issues with mongodb randomly cashing, the last crash the logs displayed a segfault with a backtrace. usually they just said:

Fri Dec 10 08:52:50 [conn449] Uncaught std::exception: std::bad_alloc, terminating

The site in question is in production so this is starting to look bad, I have attached the log of the last crash. If I could get some help fixing this issues that would be grate, if you need any more information let me know.

Thanks,
Matt



 Comments   
Comment by Matthew Ford [ 06/Jan/11 ]

1.7.3 along with some query optimisations seems to have solved the problem, can close this for now - haven't seen the issue for a long time.

Comment by Matthew Ford [ 19/Dec/10 ]

running on 1.7.3 for 6 days now - not had any crashes since but will keep this ticket updated if it occurs again

Comment by Matthew Ford [ 13/Dec/10 ]

every few days - sometimes twice a day (depending on load). I'll try 1.7.3

Comment by Eliot Horowitz (Inactive) [ 13/Dec/10 ]

How often is this happening?
Can you try 1.7.3?

Comment by Matthew Ford [ 13/Dec/10 ]

Ok, sorry haven't replied sooner (was waiting for it to crash again). Attaching the new logs with vvvv on.

Comment by Scott Hernandez (Inactive) [ 10/Dec/10 ]

Yes, in the config file just add "vvv=true"

Comment by Matthew Ford [ 10/Dec/10 ]

Yes, although we did add monit at some point so there might have been some crashes when we didn't.

updates, well we do a few types:
addtoset
creating new records
upserts

The main reason I suspect that the culprit is massive or query is that it tends to be the last thing in the log before the crashes.

This is an example of what they look like:

{ "ts" : "Fri Dec 10 2010 13:32:34 GMT+0000 (BST)", "info" : "query seogadget_development.keywords scanAndOrder reslen:1824737 nscanned:10048 \nquery: { $query: { campaign_id: ObjectId('4d0117374b3e897d5a000001'), $or: [ { search_index:

{ $all: [ \"from\" ] }

}, { search_index:

{ $all: [ \"to\" ] }

}, { search_index:

{ $all: [ \"london\" ] }

}, { search_index:

{ $all: [ \"pari\" ] }

}, { search_index:

{ $all: [ \"dover\" ] }

}, { search_index:

{ $all: [ \"franc\" ] }

}, { search_index:

{ $all: [ \"disneyland\" ] }

}, { search_index:

{ $all: [ \"hotel\" ] }

}, { search_index:

{ $all: [ \"euro\" ] }

}, { search_index:

{ $all: [ \"eurostar\" ] }

}, { search_index:

{ $all: [ \"uk\" ] }

}, { search_index:

{ $all: [ \"franc\" ] }

}, { search_index:

{ $all: [ \"tunnel\" ] }

} ], hidden:

{ $ne: true }

}, $orderby:

{ search_volume: -1, visits: -1 }

} nreturned:3947 1237ms", "millis" : 1237 }

Currently in the process of refactoring our code to avoid doing this as much as possible as it's also very slow.

Is there a config option for -vvvv so I don't have to run the process in screen.

Comment by Eliot Horowitz (Inactive) [ 10/Dec/10 ]

Did you do a --repair after the first crash?
What kind of updates are you doing?
Can you run with -vvvv

Comment by Matthew Ford [ 10/Dec/10 ]

I have a feeling it could be that some of our queries use '$or' with many conjunctions

Generated at Thu Feb 08 02:59:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.