For a sharded setup there are stale config version issues when read preference primary is used.
This problem appears to go away when read preference is removed. The logs when this problem happens indicate that when a stale config is detected mongos is using a different connection when calling setShardVersion and then doing the query.