[SERVER-17681] Secondary freeze while sync Created: 21/Mar/15 Updated: 23/Mar/15 Resolved: 23/Mar/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tal Rasha | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
when there is a lot of inserts(about 4Kops/s), it happens randomly, i think there's a deadlock. Here's the state: rs1|_id:1's cpu usage is 0, and it will sync as expected if I restart it.
mongod is started by: |
| Comments |
| Comment by Sam Kleinman (Inactive) [ 23/Mar/15 ] |
|
Thanks for this information. MongoDB+MMAP did not (typically) use the full CPU and IO capacity of a system, and as a result many deployments used an approach with multiple mongod instances on a single host connected to RAID arrays with high-throughput in order to use resources more efficiently. MongoDB+WiredTiger in 3.0, is, however, able to effectively use more CPU and IO resources because of it's overall increased efficiency and compression features: as a result, running multiple mongod instances on a single host may cause a situation where the mongod instances are contending for the same resources. In one of these situations we would expect to see exactly what you describe here: some mongod instances will appear to freeze when the operating system schedules other processes. I would recommend lowering the number of mongod instances on each host until the observed contention rate during initial sync is less that the total capacity of the system. I don't see evidence of a bug, so I'm going to close this ticket because the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. Regards, |
| Comment by Tal Rasha [ 23/Mar/15 ] |
|
mongod --version OS: CentOS 6.5 There are 3 same machines, each of them has a config server, a mongos instance, 4 mongod instances. It seems there's no useful log |
| Comment by Asya Kamsky [ 21/Mar/15 ] |
|
Could you provide the exact version you are running? (mongod --version ought to do it). At the time you observe this, what is in the logs on the secondaries that "freeze"? Can you provide the log or at least the last dozen or so lines? How much RAM is on the machine this happens on? (I see you specify 8GB for wired tiger cache, out of how many is that?) |