Summary:

If a user creates a special index type and then restarts with a version of mongod that doesn't support this index type, mongod will assume the index is a regular ascending index and add invalid entries. These invalid entries can persist even if the document gets removed (e.g. after re-upgrading). This can lead to data corruption if an invalid entry is used as a target for an update, all without ever returning an error to the client.

Repro:

In this example, a user maintains a collection of documents with location data. The user starts with 2.4, makes a 2dsphere index, and performs a series of downgrades and upgrades. Eventually, the user tries to label each document in the collection with its respective country.

mongodb-osx-x86_64-2.4.0-rc1/bin/mongod --fork --logpath mongod.log 
sleep 1
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.ensureIndex({geo:"2dsphere"})'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.2.3/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.insert({geo:{type:"Point",coordinates:[40,40]}})'
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.4.0-rc1/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.remove()'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.insert({geo:{type:"Point",coordinates:[30,30]}})'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.2.3/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.update({geo:{type:"Point",coordinates:[30,30]}},{$set:{country:"Egypt"}})'
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.update({geo:{type:"Point",coordinates:[40,40]}},{$set:{country:"Turkey"}})'

The inserts/remove/updates above all return success. But, this document gets the wrong country; [30,30] is Egypt, not Turkey:

> db.foo.findOne()
{
	"_id" : ObjectId("512d7b5bf5de9b6b5b926c9e"),
	"country" : "Turkey",
	"geo" : {
		"type" : "Point",
		"coordinates" : [
			30,
			30
		]
	}
}
>

Logfile attached.

Walkthrough of the repro above:

• Line 3: having started up 2.4, the user creates a 2dsphere index.
• Line 7: after restarting in 2.2, the user inserts a document with coords [40,40]. Not recognizing the index type, the server generates a log message "warning: can't find plugin [2dsphere]", but then goes on to operate on the 2dsphere index as if it were a regular ascending index. Thus, the document gets successfully inserted into the collection, and the 2dsphere index gets a new entry in an invalid format.
• Line 11: after restarting in 2.4, the user removes the document just inserted. The server is unable to find the document's 2dsphere index entry and generates a log message "unindex failed (key too big?) test.foo.$geo_2dsphere key: { : "2f00031333311103" } _id: ObjectId('512d952eeeaa3bfbf4a1262b')", but removes the document from the collection, leaving the invalid index entry in place.
• Line 12: immediately after, the user inserts a new document with coords [30,30]. This new document happens to get the same diskloc as the old document (since the newly-freed slot was put back on the head of the appropriate freelist), so the old invalid index entry is now pointing to this document.
• Line 16: after restarting in 2.2, the user performs an update, hoping to target the newly-inserted document (coords [30,30]) to have country "Egypt". The update uses a covered query on the 2dsphere index to find the document to target, again mistaking it for an ascending index. Not finding any matches in the index, the server returns from the operation having updated no documents.
• Line 17: immediately after, the user performs another update, this one targeting documents with coords [40,40] to have country "Turkey". The update again uses a covered index query for the filter. The server notices the 2dsphere index entry created in 2.4 but ignores it, logging "unindex failed (key too big?) test.foo.$geo_2dsphere key: { : { type: "Point", coordinates: [ 30.0, 30.0 ] } } _id: ObjectId('512d952f634d9e41fd626f50')". Instead, the update finds the invalid index entry created for the first document, which points to the same diskloc as the second document. Thus, the document in the collection with coords [30,30] is targeted for the update meant for documents with coords [40,40].