[SERVER-8242] assertion failures (btree) during repair Created: 18/Jan/13 Updated: 10/Dec/14 Resolved: 03/Jun/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Zac Witte | Assignee: | Unassigned |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
ubuntu on EC2 |
||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Was running a repair with the following command and got this assertion error in the log of 1 out of 3 shards. The other shards had a different assertion error, which I filed a different bug for: ubuntu@mongo3:~$ mongod --version sudo -u mongodb mongod --dbpath=/db/mongodb --repair --repairpath=/export/mongodb --nojournal Fri Jan 18 03:25:59 [initandlisten] 47127700/861940368 5% |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 03/Jun/13 ] |
|
Looks to be the same problem as |
| Comment by Aaron Staple [ 21/Mar/13 ] |
|
Hi Zac, One other thing to be aware of is that the temporary external sort files go inside _tmp within your dbpath not your repair path. So a disk error including out of disk space on the partition containing /db/mongodb could cause this issue. We have to improve reporting when there is an error during a repair operation. For this ticket and |
| Comment by Zac Witte [ 24/Jan/13 ] |
|
OK so it sounds like mongod needs to be more robust in how it reads in files and retries on failure. At the very least it needs more user-friendly error messages. |
| Comment by Aaron Staple [ 23/Jan/13 ] |
|
Hi Zac, Here's some analysis of the two assertions described in the log. But first some background info: A foreground index build is implemented in three phases: 1) The whole collection is scanned, and for each document in the collection the set of keys that will appear in the index are extracted from the document. The keys for all documents, along with the locations of these documents, are saved into a set of temporary external sort files in a temporary directory ( named _tmp ) within your dbpath. (The keys within each external sort file are sorted, but there is no sorting relationship between the different files.) 2) A merge sort over the external sort files is used to iterate over all the keys in the external sort files, in order. The keys are used to build the leaves of a new btree. 3) The internal nodes of the btree are built up based on the leaves of the btree. The first error in the log, about "la <= 0xffffff" is occurring because an invalid document location is being passed from an external sort file to a new btree leaf as part of step 2. The second error in the log, about "reading doc for external sort failed" describes an error reading an external sort file. It looks like it's occurring while reading an index key from the external sort file. It looks like part of the key is read but there is an error while reading the rest of the key. I believe what's happening is that the implementation assumes the first error is a dup key error, even though it seems to result from a bad read of the external sort file. Then it continues trying to read from the external sort file until it sees the second error and at that point fails. |
| Comment by Zac Witte [ 22/Jan/13 ] |
|
I no longer have the full logs, but I have the excerpts where the exceptions occurred. I'll attach them here. I think everything looked normal until this exception. The /export partition did not run out of disk space, but it is possible that the EBS volumes or the network connection had a hiccup. That kind of thing is always possible with EC2. It is strange that all 3 servers had these assertion errors at different times, though. I noticed this in my syslog but I can't be sure if it's related since there's no way to match timestamps: [11797218.035099] md/raid0:md100: md_size is 10737408000 sectors. |
| Comment by Aaron Staple [ 22/Jan/13 ] |
|
Hi Zac - Can you send the full log? Also, is it possible there was a disk error or you ran out of disk space while the repair was running? |