[SERVER-22946] MongoDB crashing Created: 03/Mar/16 Updated: 15/Mar/16 Resolved: 15/Mar/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 3.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Aaron | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | Try to add new record to a big database(20GB), and it keep crashing. |
| Participants: |
| Description |
|
It is keep crashing when add new record,here is the Traceback:
|
| Comments |
| Comment by Kelsey Schubert [ 08/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi liuaaronsz, Thank you for confirming that you have a corrupt index. However, it is challenging to determine whether the corruption is isolated to the index. Therefore, I would still recommend executing repairDatabase. Kind regards, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron [ 08/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||
On Tue, Mar 8, 2016 at 12:02 PM, Thomas Schubert (JIRA) <jira@mongodb.org> | |||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 08/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi liuaaronsz, Unfortunately, the behavior you are describing could indicate that your backups have the same corruption issue.
To check the integrity of your database files, please consider executing db.collection.validate(true) on the affected collection and attaching the output to this ticket. Thank you, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Aaron [ 08/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi Thomas Here are the answers, and we are not sure the file corruption caused this
Is there some tool you have to check the integrity of database file? On Mon, Mar 7, 2016 at 3:53 PM, Thomas Schubert (JIRA) <jira@mongodb.org> | |||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 07/Mar/16 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi liuaaronsz, The stack trace you have provided indicates that this node has some data files that have become corrupt in some way. It is not clear if the corruption resides in the index or the data itself. In cases like this, it is very challenging to determine whether the corruption is isolated beyond the file level. I have compiled a list of routine questions about data storage and the configuration of your environment. We can use these questions to help get a better understanding of what is going on here. Often corruption is the result of faulty disk drives or power failures. But, please note that in these sorts of situations it can be difficult to understand the cause of the corruption without a straightforward reproduction.
I would recommend a clean resync from a node that is not affected. If that is not possible, I would recommend executing repairDatabase. Before attempting repairDatabase, please consider backing up your files. I would also suggest that you check the integrity of the affected nodes's disk drives. If this issue persists, you may need to replace them. Regards, |