When running the attached TestCase, I'll not see any outputs due to the fact that findAndModify Operations are run from the $cmd Collection. The corresponding DBApiLayer.MyCollection.__find(DBObject, DBObject, int, int, int, int, ReadPreference, DBDecoder) call is accessing OutMessage.query without handling a DBEncoder, so the driver falls back to the defaultEncoder ignoring custom set Encoders through db.getCollection("$cmd").setDBEncoderFactory(). From my point of view the driver should be set per Default to a given Encoder/Decoder(-Factory), which then can be overriden with a custom Encoder/Decoder(-Factory).

Attached is as well a patch fixing this specific mentioned problem, but other locations referencing DB(En|De)coder.FACTORY should be considered for changes as well.