[SERVER-14213] Config Server Corruption - BSONObj size: 1852404841 (0x6974696E) is invalid. Size must be between 0 and 16793600(16MB) Created: 09/Jun/14 Updated: 10/Dec/14 Resolved: 09/Jun/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Mike | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
Hi MongoDB, I'm running a sharded cluster with:
When I attempt to shard a new collection on an existing DB I get the following:
I'm also seeing the following logged on each config server:
as well as entries like:
I believe my config server collections (lockpings and mongos) have bad data in them... in fact when I look at the documents in each there are old mongos entries that don't exist and there inconsistent lock times or entries that are valid when comparing across the 3 config servers Any idea on how to resolve this? It's a production instance so I'm hesitant to make a change and it doesn't sound like my config backups will help since this has been going on past the retention threshold I have... Thanks so much! |
| Comments |
| Comment by Ramon Fernandez Marina [ 09/Jun/14 ] |
|
Hi amarettoslim, as you point out, the issue came from one config server having corrupted data. Corruption can happen for various reasons and usually is hard to track down its cause, although the more common causes are network problems or failing hard drives. I would recommend you check the health of the hard drives in the failed config server to be on the safe side. Regards, |
| Comment by Mike [ 09/Jun/14 ] |
|
I tracked the issue down to one config server instance out of the 3 and resolved the matter by replacing the bad config server's data directory with a working copy from one of the other two. http://docs.mongodb.org/manual/tutorial/replace-config-server/ I hope this helps someone else. It would still be nice to know what causes this in the first place so if someone from Mongo wants to comment that'd be excellent. -Mike |