[SERVER-2203] Mongodb keep crashing/segfaulting Created: 10/Dec/10 Updated: 12/Jul/16 Resolved: 06/Jan/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 1.6.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Matthew Ford | Assignee: | Eliot Horowitz (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
EC2 large instance (ubuntu) |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
For the last 2-3 days I have had recurring issues with mongodb randomly cashing, the last crash the logs displayed a segfault with a backtrace. usually they just said: Fri Dec 10 08:52:50 [conn449] Uncaught std::exception: std::bad_alloc, terminating The site in question is in production so this is starting to look bad, I have attached the log of the last crash. If I could get some help fixing this issues that would be grate, if you need any more information let me know. Thanks, |
| Comments |
| Comment by Matthew Ford [ 06/Jan/11 ] |
|
1.7.3 along with some query optimisations seems to have solved the problem, can close this for now - haven't seen the issue for a long time. |
| Comment by Matthew Ford [ 19/Dec/10 ] |
|
running on 1.7.3 for 6 days now - not had any crashes since but will keep this ticket updated if it occurs again |
| Comment by Matthew Ford [ 13/Dec/10 ] |
|
every few days - sometimes twice a day (depending on load). I'll try 1.7.3 |
| Comment by Eliot Horowitz (Inactive) [ 13/Dec/10 ] |
|
How often is this happening? |
| Comment by Matthew Ford [ 13/Dec/10 ] |
|
Ok, sorry haven't replied sooner (was waiting for it to crash again). Attaching the new logs with vvvv on. |
| Comment by Scott Hernandez (Inactive) [ 10/Dec/10 ] |
|
Yes, in the config file just add "vvv=true" |
| Comment by Matthew Ford [ 10/Dec/10 ] |
|
Yes, although we did add monit at some point so there might have been some crashes when we didn't. updates, well we do a few types: The main reason I suspect that the culprit is massive or query is that it tends to be the last thing in the log before the crashes. This is an example of what they look like: { "ts" : "Fri Dec 10 2010 13:32:34 GMT+0000 (BST)", "info" : "query seogadget_development.keywords scanAndOrder reslen:1824737 nscanned:10048 \nquery: { $query: { campaign_id: ObjectId('4d0117374b3e897d5a000001'), $or: [ { search_index: { $all: [ \"from\" ] }}, { search_index: { $all: [ \"to\" ] }}, { search_index: { $all: [ \"london\" ] }}, { search_index: { $all: [ \"pari\" ] }}, { search_index: { $all: [ \"dover\" ] }}, { search_index: { $all: [ \"franc\" ] }}, { search_index: { $all: [ \"disneyland\" ] }}, { search_index: { $all: [ \"hotel\" ] }}, { search_index: { $all: [ \"euro\" ] }}, { search_index: { $all: [ \"eurostar\" ] }}, { search_index: { $all: [ \"uk\" ] }}, { search_index: { $all: [ \"franc\" ] }}, { search_index: { $all: [ \"tunnel\" ] }} ], hidden: { $ne: true }}, $orderby: { search_volume: -1, visits: -1 }} nreturned:3947 1237ms", "millis" : 1237 } Currently in the process of refactoring our code to avoid doing this as much as possible as it's also very slow. Is there a config option for -vvvv so I don't have to run the process in screen. |
| Comment by Eliot Horowitz (Inactive) [ 10/Dec/10 ] |
|
Did you do a --repair after the first crash? |
| Comment by Matthew Ford [ 10/Dec/10 ] |
|
I have a feeling it could be that some of our queries use '$or' with many conjunctions |