In addition to a SyncClusterConnection, I suggest implementing TwoPhaseCommit connection (possibly long-term).
The SyncClusterConnection is a nice idea, but doesn't scale in practice: when you have lots of clients performing writes via a SyncClusterConnection, you end up flushing your mongo nodes very often. Each server only has a limited amount of flushes it can do per second, which reduces performance a lot.
Another way to achieve some sort of durability/consistency guarantee with multiple servers is two-phase commit. The flow would be as follows:
- client issues a "request update" command to all servers;
- all nodes verify this write wouldn't introduce any errors ("Update-if-Current" comes to mind);
- client receives verification from all nodes and send commit to all nodes;
- if some nodes fail during commit, eventual consistency can be used.
This does require some sort of row-level locking mechanism within the core servers, so multiple updates for the same objects will be synchronized between servers. However, this would scale very well with many clients talking to a single replication set.