<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:20:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9388] Creating large arrays of embedded documents causes blocking on secondaries.</title>
                <link>https://jira.mongodb.org/browse/SERVER-9388</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When adding large arrays of embedded documents replication becomes blocked on secondaries. This is causing queues on the slave and timouts to slaveOk() queries comming from a php driver.&lt;/p&gt;

&lt;p&gt;I am wondering if this could be related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-2771&quot; title=&quot;Background index builds on replica set secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-2771&quot;&gt;&lt;del&gt;SERVER-2771&lt;/del&gt;&lt;/a&gt; as I read somewhere that indexes are created on arrays. I could not find any corroborating information. &lt;/p&gt;</description>
                <environment>linux</environment>
        <key id="72223">SERVER-9388</key>
            <summary>Creating large arrays of embedded documents causes blocking on secondaries.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="stephen.steneker@mongodb.com">Stennie Steneker</assignee>
                                    <reporter username="robert.jobson@dominionenterprises.com">Robert Jobson</reporter>
                        <labels>
                    </labels>
                <created>Thu, 18 Apr 2013 14:41:28 +0000</created>
                <updated>Wed, 10 Dec 2014 23:11:39 +0000</updated>
                            <resolved>Wed, 24 Apr 2013 02:30:13 +0000</resolved>
                                    <version>2.2.3</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="322454" author="robert.jobson@dominionenterprises.com" created="Fri, 26 Apr 2013 18:21:37 +0000"  >&lt;p&gt;This was more of a worst case scenerio to duplicate the issue. What we are actually doing is adding city state pairs to new documents as a batch job that only runs once a week in the mongo js shell. As suggested I&apos;ll work on making the process cleaner on the other end as a work around. &lt;/p&gt;

&lt;p&gt;Clearly as stated above when writing up the issue I believed this to be a bug. This was primarily supported by not seeing any resource bottlenecks on the server statistics during the weekly batch and not encoutering similar issues any other time including more resource intensive processes. &lt;/p&gt;

&lt;p&gt;This lead me to believe there was some kind of blocking issue. Further...(during multithreaded replication batch operations)&quot;To ensure this consistency, MongoDB blocks all read operations while applying the batch of operations.&quot; I suspect the replication batching operation may be written such that it is taking too large a chunk when presented with this scenerio and that is the core of the delays.&lt;/p&gt;
</comment>
                            <comment id="320106" author="stennie" created="Wed, 24 Apr 2013 02:29:56 +0000"  >&lt;p&gt;Hi Robert,&lt;/p&gt;

&lt;p&gt;The main issue here is likely that your successive array pushes are creating a large amount of data to replicate, and overwhelming your servers&apos; ability to keep up with replication.  Replicated operations are converted into an &lt;a href=&quot;http://docs.mongodb.org/manual/reference/glossary/#term-idempotent&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;idempotent&lt;/a&gt; form in the replication oplog. The idempotent form allows the same operation to be applied safely multiple times if needed during synchronization.&lt;/p&gt;

&lt;p&gt;In the case of arrays, your `$push` operation on the primary is converted into an oplog `$set` entry with the array contents after applying the push.  After each loop iteration you will have a progressively larger array to set and replicate.  Substantially growing your document quickly will also have consequences on disk space allocation, and may lead to excessive disk space usage and I/O as the document may require frequent moves to a larger allocation within the data files.&lt;/p&gt;

&lt;p&gt;If you are actually planning on doing 10,000 push updates in short succession on the same document, a better approach would be to apply these updates on the client side to minimize the repetitive pushes. Alternatively, you should consider flattening your schema rather than using a large embedded array.  Additionally, you should consider that the secondaries have to write as much data as the primaries so use of slaveOK will not necessarily improve your read performance.&lt;/p&gt;

&lt;p&gt;Lastly, please keep in mind that the SERVER project is intended for reporting bugs and feature suggestions.  For discussion of use cases such as this, you would be best posting a discussion on the mongodb-users group (&lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://groups.google.com/group/mongodb-user&lt;/a&gt;) or a specific question on Stack Overflow / ServerFault.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Stephen&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 24 Apr 2013 02:29:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 42 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 42 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>robert.jobson@dominionenterprises.com</customfieldvalue>
            <customfieldvalue>stephen.steneker@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmwmv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrncj3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>49980</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;use test;&lt;br/&gt;
for(t = 0; t &amp;lt; 10000; t++){&lt;br/&gt;
db.embedded_doc.insert({_id :t});&lt;br/&gt;
for(i = 0; i &amp;lt; 10000; i++){&lt;br/&gt;
	db.embedded_doc.update({_id : t}, { $push : { test_array : &lt;/p&gt;
{ field1 : &quot;VALUE&quot;, field2 : i }
&lt;p&gt; } });&lt;br/&gt;
}&lt;br/&gt;
}&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0nrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>