[SERVER-2783] Traceback of crash in mongod Created: 17/Mar/11 Updated: 30/Mar/12 Resolved: 14/Jun/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 1.8.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Vince Busam | Assignee: | Aaron Staple |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 9.10, 64bit, 10gen packages |
||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Two separate continuous chunks of the logfile with tracebacks after mongod (as a shard) crashed. |
| Comments |
| Comment by Aaron Staple [ 14/Jun/11 ] |
|
Hi Vince, Are there any pending issues for this ticket? |
| Comment by Aaron Staple [ 29/Mar/11 ] |
|
Glad you were able to regenerate the collection. Just fyi, the mongodump issue could potentially be related to the bad index unless you forced deletion of your _id indexes. |
| Comment by Vince Busam [ 29/Mar/11 ] |
|
I dumped and re-created the collection on the shard, which solved this. mongod/mongodump crashed during dumping the collection (after the indexes were dropped), so it may have been corrupt database files. |
| Comment by Aaron Staple [ 22/Mar/11 ] |
|
I'm seeing a few different types of stack traces in the most recently supplied log, all of which may be caused by a bad index entry. The bad index could have resulted from a bad shutdown or a bug in 1.6 related to large keys that is sometimes but not always accompanied by log messages indicating 'key too large' in the log file. I would recommend rebuilding your indexes. |
| Comment by Aaron Staple [ 22/Mar/11 ] |
|
Have you had any crashes on this system besides those in the log file? (You mentioned that you reimported your data once.) |
| Comment by Aaron Staple [ 22/Mar/11 ] |
|
Sorry, from log it looks like no durability. |
| Comment by Aaron Staple [ 22/Mar/11 ] |
|
Ok thanks. Are you running with durability? |
| Comment by Vince Busam [ 22/Mar/11 ] |
|
Oops, bad grep. Here's the full log. |
| Comment by Vince Busam [ 22/Mar/11 ] |
|
Fixed full log. |
| Comment by Aaron Staple [ 22/Mar/11 ] |
|
Hi Vince - the log that was sent doesn't seem to contain actual backtraces in it. Where did the backtraces in the log excerpts you initially included in this bug report come from? |
| Comment by Vince Busam [ 21/Mar/11 ] |
|
Full log from 2 days before crash. |
| Comment by Vince Busam [ 21/Mar/11 ] |
They go back to installing mongo. No mention of key too large.
I'll cut out a few days and upload the file in a bit.
Originally, they were created with background=true after data was present, but then the whole collection was dumped and re-imported, and the indexes were created before the import.
I found another one that adds a geocoords box to the other count(): , instanceslist: 33839, geocoords.lat: { $lte: 41.44272637767212, $gt e: 21.45306863308678 }} } reslen:64 283ms
Yes. |
| Comment by Aaron Staple [ 21/Mar/11 ] |
|
Thanks for the replies. Just a few followup questions:
|
| Comment by Vince Busam [ 21/Mar/11 ] |
|
The database and indexes were created in 1.6.5. Following are the stats (with indexes) and an example document. Our count operations are: ) No sign of "unindex: key too large" in the logs. > db.globalusers.stats() , , > db.globalusers.findOne( {"lowerscreenname":"vincebusam"}) }, }, } , , }, , , |
| Comment by Vince Busam [ 21/Mar/11 ] |
|
addr2line of mongocrash:
From mongocrash2:
|
| Comment by Aaron Staple [ 21/Mar/11 ] |
|
Vince - also do you have any messages "unindex: key too large" in the logs? |
| Comment by Aaron Staple [ 21/Mar/11 ] |
|
Hi Vince - were you using data files and indexes from an earlier mongo version? Can you send your indexes and an example document for this collection as well as your count query? Thanks. |
| Comment by Aaron Staple [ 19/Mar/11 ] |
|
Hi Vince - could you let us know the count queries you are sending? |
| Comment by Eliot Horowitz (Inactive) [ 19/Mar/11 ] |
|
Can't right now but you can get the symbols from: http://downloads.mongodb.org/linux/mongodb-linux-x86_64-debugsymbols-1.8.0.tgz and then use addr2line |