[SERVER-14676] Index creation caused Fatal Assertion 16967 Created: 24/Jul/14 Updated: 16/Nov/21 Resolved: 12/Aug/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance, Storage |
| Affects Version/s: | 2.6.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ryan Carmichael | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | ALL | ||||
| Participants: | |||||
| Description |
|
On OSX, with mongo version: Single server, no replica sets. Mongo crashed when I tried to ensureIndex on a collection. I had previously (as in a few minutes before) successfully added other indexes. Mongo starts back up but db.repairDatabase() crashes it again. It is a relatively small/medium sized collection - about 11 million rows at a taking up about 10GB of disk space. Snippet below (see full stack traces attached)
|
| Comments |
| Comment by Ramon Fernandez Marina [ 12/Aug/14 ] |
|
Thanks for the update ryan.carmichael@elabsolutions.com, glad to hear 2.6.3 is working for you. We're going to mark this ticket as resolved, since not having database files makes it impossible to do any further investigation. Regards, |
| Comment by Ryan Carmichael [ 12/Aug/14 ] |
|
Just as a sort of final comment on my side, I updated to 2.6.3, wiped the database and did bunch more inserts with the same data, ensureIndexes, etc., etc., all with logLevel set to 1. Everything was A-OK, no repeats of the crash, repairDb runs with no issues. Removed all the data, inserted the 11M rows again, several times. All good. |
| Comment by Ramon Fernandez Marina [ 01/Aug/14 ] |
|
Thanks for the update rcarmich. I'm going to keep this open a tad longer and revisit the stack traces to see if we can find something. One suggestion for your consideration is to increase the logLevel to 1 if you re-insert the data. If there's a bug in MongoDB and you hit the problem again, the verbose logs may help us find out what's going on. Thanks, |
| Comment by Ryan Carmichael [ 01/Aug/14 ] |
|
Unfortunately I have not been able to easily reproduce this again. I suppose the ticket could be closed. If this were a production system I would be able to spend more time figuring it out but since it is a development machine it will be easier to just wipe the data. Thanks for looking into it. |
| Comment by Ryan Carmichael [ 29/Jul/14 ] |
|
This is the type of data inserted (about 11 million rows of it). The data below is fake, not real patient info. { _id: ObjectId('53d150edd6c4a034594b1eb7'), action: "Added medication 'XANAX'", objectType: "com.xyz.abc.Specimen", searchParams: "Jim|Dr. XYZ|111223X|Urine|123-555-1212|St. Louis|MO|St. Louis|J234382|Smith|111_111223X|", username: "myusername", eventId: 16, eventTime: new Date(1402423097000), objectId: 529237 }The indexes I had were on: The index I was trying to add when it crashed: I was adding them via: db.actionLog.ensureIndex( {username: 1}) ) ) I am not aware of any unclean shutdowns.. Couldn't find any indication of such in the log files, except for the ones I had when the issue started and it crashed during the index build. The SMART status of my hard drive is fine. I will try to run a full diagnostic of it tomorrow. |
| Comment by Ramon Fernandez Marina [ 28/Jul/14 ] |
|
rcarmich, did you have any unclean shutdowns before the problems started? Also, can you tell us a bit more about the number of indexes and their shape? Also, how were you building those indexes? Finally, can you double-check the health of your hard drives? Flaky drives are a common cause for data corruption, and we need to rule that option out no matter how unlikely. I'm trying to reproduce on my end with sample documents containing numbers, strings, and arrays of numbers. If you manage to reproduce the problem with data you can share please let us know. Thanks, |
| Comment by Ryan Carmichael [ 26/Jul/14 ] |
|
Unfortunately the data contains patient information and is not something I can share. It probably doesn't help any but what preceded this issue was me inserting a bunch of data into the table, creating some indexes, and then db.actionLog.remove({}). I loaded and erased the data probably 6 or 7 times, doing db.repairDatabase() several times in between. Basically just messing around with converting some mysql data into mongo. On Monday I will attempt to recreate the issue with non sensitive data, but I expect it will be difficult to reproduce. In this case the loss or corruption of data isn't important because it was on a development machine... Just have to convince my boss this won't happen in production |
| Comment by Ramon Fernandez Marina [ 25/Jul/14 ] |
|
rcarmich, looks like your indexes are corrupted, but I would not expect db.repairDatabase() to cause a crash. Is there a chance we can get a copy of your data so we can investigate this problem? If this is an option, we can provide a way to upload the data securely and confidentially – please let us know. Thanks, |