[SERVER-9388] Creating large arrays of embedded documents causes blocking on secondaries. Created: 18/Apr/13  Updated: 10/Dec/14  Resolved: 24/Apr/13

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Robert Jobson Assignee: Stennie Steneker (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux


Operating System: ALL
Steps To Reproduce:

use test;
for(t = 0; t < 10000; t++){
db.embedded_doc.insert({_id :t});
for(i = 0; i < 10000; i++){
db.embedded_doc.update({_id : t}, { $push : { test_array :

{ field1 : "VALUE", field2 : i }

} });
}
}

Participants:

 Description   

When adding large arrays of embedded documents replication becomes blocked on secondaries. This is causing queues on the slave and timouts to slaveOk() queries comming from a php driver.

I am wondering if this could be related to SERVER-2771 as I read somewhere that indexes are created on arrays. I could not find any corroborating information.



 Comments   
Comment by Robert Jobson [ 26/Apr/13 ]

This was more of a worst case scenerio to duplicate the issue. What we are actually doing is adding city state pairs to new documents as a batch job that only runs once a week in the mongo js shell. As suggested I'll work on making the process cleaner on the other end as a work around.

Clearly as stated above when writing up the issue I believed this to be a bug. This was primarily supported by not seeing any resource bottlenecks on the server statistics during the weekly batch and not encoutering similar issues any other time including more resource intensive processes.

This lead me to believe there was some kind of blocking issue. Further...(during multithreaded replication batch operations)"To ensure this consistency, MongoDB blocks all read operations while applying the batch of operations." I suspect the replication batching operation may be written such that it is taking too large a chunk when presented with this scenerio and that is the core of the delays.

Comment by Stennie Steneker (Inactive) [ 24/Apr/13 ]

Hi Robert,

The main issue here is likely that your successive array pushes are creating a large amount of data to replicate, and overwhelming your servers' ability to keep up with replication. Replicated operations are converted into an idempotent form in the replication oplog. The idempotent form allows the same operation to be applied safely multiple times if needed during synchronization.

In the case of arrays, your `$push` operation on the primary is converted into an oplog `$set` entry with the array contents after applying the push. After each loop iteration you will have a progressively larger array to set and replicate. Substantially growing your document quickly will also have consequences on disk space allocation, and may lead to excessive disk space usage and I/O as the document may require frequent moves to a larger allocation within the data files.

If you are actually planning on doing 10,000 push updates in short succession on the same document, a better approach would be to apply these updates on the client side to minimize the repetitive pushes. Alternatively, you should consider flattening your schema rather than using a large embedded array. Additionally, you should consider that the secondaries have to write as much data as the primaries so use of slaveOK will not necessarily improve your read performance.

Lastly, please keep in mind that the SERVER project is intended for reporting bugs and feature suggestions. For discussion of use cases such as this, you would be best posting a discussion on the mongodb-users group (http://groups.google.com/group/mongodb-user) or a specific question on Stack Overflow / ServerFault.

Thanks,
Stephen

Generated at Thu Feb 08 03:20:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.