Our current master db server had no enough RAM, so I tried manually switch one of the slave to master.
After stopping update master db, and wait for sufficient time to got slave catch up. I checked the count of the collections, but found that there's minor loss of documents. ( master: 99344270; slave 1: 99344259, 11 docs lost; slave 2: 99344251, 19 docs lost.). It's unacceptable for a DB that loss data.

Following is the stats:

• master (after stop doing updates)

> db.printReplicationInfo()
configured oplog size: 25000MB
log length start to end: 502311secs (139.53hrs)
oplog first event time: Wed Jun 29 2011 19:12:30 GMT+0800 (CST)
oplog last event time: Tue Jul 05 2011 14:44:21 GMT+0800 (CST)

> db.my_collection.count()
99344270

• on slave 1

> db.my_collection.count()
99344259

*on slave 2
> db.my_collection.count()
99344251