[SERVER-8751] Downgrading 2.4=>2.2 without dropping 2dsphere/text indexes can lead to data corruption Created: 27/Feb/13  Updated: 07/Mar/14  Resolved: 05/Mar/13

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.4.0-rc1
Fix Version/s: 2.4.0-rc2

Type: Bug Priority: Major - P3
Reporter: J Rassi Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Issue Links:
Depends
depends on SERVER-8796 Set index plugin and version on index... Closed
is depended on by DOCS-1188 Documentation for index type/plugin k... Closed
Related
related to SERVER-5826 Creating an index using a non-existin... Closed
related to SERVER-13105 2.6 prevents creation of new 2d/hashe... Closed
related to SERVER-8847 Implement dynamic storage format vers... Closed
is related to SERVER-8921 improve error message on invalid text... Closed
is related to SERVER-8920 Add new fields to dbStats command for... Closed
is related to SERVER-8923 Users with {foo:"1"} indexes able to ... Closed
Backwards Compatibility: Major Change
Operating System: ALL
Participants:

 Description   

Summary:

If a user creates a special index type and then restarts with a version of mongod that doesn't support this index type, mongod will assume the index is a regular ascending index and add invalid entries. These invalid entries can persist even if the document gets removed (e.g. after re-upgrading). This can lead to data corruption if an invalid entry is used as a target for an update, all without ever returning an error to the client.

Repro:

In this example, a user maintains a collection of documents with location data. The user starts with 2.4, makes a 2dsphere index, and performs a series of downgrades and upgrades. Eventually, the user tries to label each document in the collection with its respective country.

mongodb-osx-x86_64-2.4.0-rc1/bin/mongod --fork --logpath mongod.log 
sleep 1
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.ensureIndex({geo:"2dsphere"})'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.2.3/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.insert({geo:{type:"Point",coordinates:[40,40]}})'
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.4.0-rc1/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.remove()'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.foo.insert({geo:{type:"Point",coordinates:[30,30]}})'
mongodb-osx-x86_64-2.4.0-rc1/bin/mongo --eval 'db.adminCommand({shutdown:1})' || true
sleep 1
mongodb-osx-x86_64-2.2.3/bin/mongod --fork --logpath mongod.log --logappend
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.update({geo:{type:"Point",coordinates:[30,30]}},{$set:{country:"Egypt"}})'
mongodb-osx-x86_64-2.2.3/bin/mongo --eval 'db.foo.update({geo:{type:"Point",coordinates:[40,40]}},{$set:{country:"Turkey"}})'

The inserts/remove/updates above all return success. But, this document gets the wrong country; [30,30] is Egypt, not Turkey:

> db.foo.findOne()
{
	"_id" : ObjectId("512d7b5bf5de9b6b5b926c9e"),
	"country" : "Turkey",
	"geo" : {
		"type" : "Point",
		"coordinates" : [
			30,
			30
		]
	}
}
>

Logfile attached.

Walkthrough of the repro above:

  • Line 3: having started up 2.4, the user creates a 2dsphere index.
  • Line 7: after restarting in 2.2, the user inserts a document with coords [40,40]. Not recognizing the index type, the server generates a log message "warning: can't find plugin [2dsphere]", but then goes on to operate on the 2dsphere index as if it were a regular ascending index. Thus, the document gets successfully inserted into the collection, and the 2dsphere index gets a new entry in an invalid format.
  • Line 11: after restarting in 2.4, the user removes the document just inserted. The server is unable to find the document's 2dsphere index entry and generates a log message "unindex failed (key too big?) test.foo.$geo_2dsphere key: { : "2f00031333311103" } _id: ObjectId('512d952eeeaa3bfbf4a1262b')", but removes the document from the collection, leaving the invalid index entry in place.
  • Line 12: immediately after, the user inserts a new document with coords [30,30]. This new document happens to get the same diskloc as the old document (since the newly-freed slot was put back on the head of the appropriate freelist), so the old invalid index entry is now pointing to this document.
  • Line 16: after restarting in 2.2, the user performs an update, hoping to target the newly-inserted document (coords [30,30]) to have country "Egypt". The update uses a covered query on the 2dsphere index to find the document to target, again mistaking it for an ascending index. Not finding any matches in the index, the server returns from the operation having updated no documents.
  • Line 17: immediately after, the user performs another update, this one targeting documents with coords [40,40] to have country "Turkey". The update again uses a covered index query for the filter. The server notices the 2dsphere index entry created in 2.4 but ignores it, logging "unindex failed (key too big?) test.foo.$geo_2dsphere key: { : { type: "Point", coordinates: [ 30.0, 30.0 ] } } _id: ObjectId('512d952f634d9e41fd626f50')". Instead, the update finds the invalid index entry created for the first document, which points to the same diskloc as the second document. Thus, the document in the collection with coords [30,30] is targeted for the update meant for documents with coords [40,40].


 Comments   
Comment by Mathias Stearn [ 05/Mar/13 ]

Still needs tests. Only manual testing done.

Comment by auto [ 05/Mar/13 ]

Author:

{u'date': u'2013-03-01T22:41:38Z', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-8751 Bump pdfile minor version on new index plugin creation

Changes:
1) Warn on startup if using old minor version and invalid indexes exist.
2) Treat unknown index plugins created under pre-2.4 as ascending
3) When creating first index using new plugins:
a) if there are invalid indexes error out
b) if not pump the pdfile minor version
4) ScanAndOrder no longer supports index plugins (it never should have)

Note:
All text and 2dsphere indexes created prior to this commit will need to be rebuilt.
Branch: master
https://github.com/mongodb/mongo/commit/9c7f2055e3977c988ea7b6fa3c8762cdaafe0509

Comment by J Rassi [ 27/Feb/13 ]

Note this can also be triggered in a 2.2=>2.4 upgrade, e.g. if the user attempts to create a 2dsphere/text index on 2.2, and then upgrades to 2.4 without dropping the index.

Generated at Thu Feb 08 03:18:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.