[SERVER-6904] Inner Collection (w/ > 10000 entries) with Indexed Fields Dramatically hurt performance when adding new fields on the outer document or entries in the inner collection Created: 30/Aug/12 Updated: 15/Feb/13 Resolved: 21/Jan/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.2.0-rc0, 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Gabriel Lesperance | Assignee: | Aaron Staple |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Behavior witnessed on OS X & Ubuntu 12.04 |
||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
The following has been tested on mongoDB v2.2.0rc0 & mongoDB v2.2.0:
First initialize the database: > use test ) Then populate a document's inner collection with objects: > for ( var i = 1; i < 10000; i++) { db.testdocuments.update( { _id : ObjectId("000000000000000000000001") }, { $push : { 'a' : { uid1 : i, uid2: i }} }) } Then add a new outer field: > db.testdocuments.update( { _id : ObjectId("000000000000000000000001") }, { $set : { 'aNewOuterField' : 1 }}) All is well and the update takes normal time. But, if those inner documents are indexed: > db.testdocuments.ensureIndex( { 'a.uid1' : 1 }) mongodb-console-> Thu Aug 30 14:17:31 [conn4] build index test.testdocuments { a.uid1: 1.0 } mongodb-console-> Thu Aug 30 14:17:34 [conn4] build index done. scanned 1 total records. 2.244 secs and we add another outer field : > db.testdocuments.update( { _id : ObjectId("000000000000000000000001") }, { $set : { 'anotherOuterField' : 1 }}) Then we get the MongoDB console reporting the action as slow: mongodb-console-> Thu Aug 30 14:20:01 [conn4] update test.testdocuments query: { _id: ObjectId('000000000000000000000001') }update: { $set: { anotherOuterField: 1.0 }} idhack:1 nupdated:1 keyUpdates:0 locks(micros) w:4191036 4191ms Indeed, the last operation has been reported to have taken more than 4 seconds even though there was no index modification at all. This might be due to the document being moved but setting the profiler 'on', > db.setProfilingLevel(1) and adding another outer field: > db.testdocuments.update( { _id : ObjectId("000000000000000000000001") }, { $set : { 'oneLastOuterField' : 1 }}) gives us the following: > db.system.profile.find().pretty() { , }, , }, which definitely shows that the delay isn't due to the document being 'moved'. Adding to the inner collection is not faster either as > db.testdocuments.update( { _id : ObjectId("000000000000000000000001") }, { $push : { 'a' : { uid1 : -1, uid2: -1 }} }) gives us the following profile > db.system.profile.find().pretty() { , } , }, |
| Comments |
| Comment by Gabriel Lesperance [ 21/Jan/13 ] |
|
No problem. Thanks for the support. |
| Comment by Aaron Staple [ 21/Jan/13 ] |
|
Hi Gabriel - I'm going to resolve this as a duplicate of |
| Comment by Aaron Staple [ 16/Jan/13 ] |
|
Hi Gabriel, When an update is requested that can't be accomplished by simply modifying the value of an existing field in place (think $inc modifying a field in place), and the document does not need to move to a new disk location, the update code will do the following: 1) extract all index keys for the old version of the document The slowdown you are seeing results from steps 1 and 2. After you add your a.uid index, steps 1 and 2 each need to extract 10k keys and store them in an std::set. (Before you added the index in your test, steps 1 and 2 did not need to be run for this index.) I've filed two tickets to work on relevant optimizations:
|
| Comment by Gabriel Lesperance [ 09/Jan/13 ] |
|
Any progress on this? |