Simply put, when a client is connected to a replica set and the replica set fails over from one master to another, the Mongo instance never detects the change and continues to erroneously send all commands to the old slave.

This was discussed on the mailing list here:

http://groups.google.com/group/mongodb-user/browse_thread/thread/798f269f0882a16c

Per Scott Hernandez's suggestion I have created an example of the failure that runs like as a maven test case

git://github.com/hhubris/mongo-fail-demo.git

Creating a new Mongo object works around the problem, but it doesn't feel like the right solution.

Thanks again for all the help.