[SERVER-23442] ReplicaSet Sync Failure on incorrect disk issue in WiredTiger Created: 31/Mar/16 Updated: 09/Jun/16 Resolved: 26/Apr/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, WiredTiger |
| Affects Version/s: | 3.2.4 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Chad Kreimendahl | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 0 |
| Labels: | WTplaybook | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL | |||||||||||||
| Steps To Reproduce: | 3 Existing replicaset members in a single data center.
Wait a few hours // |
|||||||||||||
| Participants: |
| Description |
|
When adding a new replicaset member to the set, while syncing across a relatively slow network connection (< 200mbps), we're seeing the replication fail with a WiredTiger error "No space left on device". However, there is substantial space left on EVERY disk on the system, including the one specifically mounted for Mongo. /dev/sda1 880G 35G 801G 5% / — a bunch of index creation messages, of which there are tens of thousands in the hours prior, followed by:
|
| Comments |
| Comment by Ramon Fernandez Marina [ 26/Apr/16 ] | |||||||||||||||||||
|
Thanks for the update sallgeud, we're closing this ticket then. Regards, | |||||||||||||||||||
| Comment by Chad Kreimendahl [ 26/Apr/16 ] | |||||||||||||||||||
|
It appears the issue was one of inodes. The old system that was upgraded was built with the assumption of few large files (because of how mmapv1 worked)... and was not reformatted with more inodes when we upgraded. We've since updated it, including moving to xfs, and ensuring there would be more than enough inodes based on what we see in our brand-new systems. | |||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 25/Apr/16 ] | |||||||||||||||||||
|
sallgeud, if I'm not mistaken the maximum number of directory entries in ext4 is 64000, so a system where the number of collections plus indexes approaches that value may behave in the manner you describe. If this is the root cause of the behavior you're seeing you can consider using XFS (which we already recommend in our production notes) or switch to directoryPerDB. Can you please count the number of collections and indexes on this deployment to see if they could be the source of the problem you're seeing? Thanks, | |||||||||||||||||||
| Comment by Chad Kreimendahl [ 31/Mar/16 ] | |||||||||||||||||||
|
1: Probably not ulimit stuff as we are a bit overkill on these numbers (for good reason):
2: potential inode limit? Might be interesting to add a note to this in your "production notes" to check that when upgrading from significantly less files (mmapv1) to significantly more smaller files (wt), you might need more inodes.
3. ext4 on the specific system with the errors.. other systems use xfs I can run it again and check inodes in the moments before the failure, as there were probably a truckload created as indexes were being copied. | |||||||||||||||||||
| Comment by Kelsey Schubert [ 31/Mar/16 ] | |||||||||||||||||||
|
Hi sallgeud, So we can continue to investigate this issue, please answer the following questions about the affected system:
Thank you, | |||||||||||||||||||
| Comment by Chad Kreimendahl [ 31/Mar/16 ] | |||||||||||||||||||
|
To save some time. This always happens on the same step, when cloning indexes. All db's had already been successfully cloned, and the failure would happen on the same indexing phase of one specific database. I won't be able to share the database itself, for contractual / privacy reasons, but I can share metadata about it, such as size, collection counts, index counts, etc: "collections" : 109, Of note is that the above stats are actually on the higher end of average, but much smaller than some databases + indexes that had already previously completed. |