<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:03:09 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-23359] WiredTiger should not cache updates between named snapshots</title>
                <link>https://jira.mongodb.org/browse/SERVER-23359</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;See the attached image. An identical workload is run against two sets (test and control) with two nodes and an arbiter each. All hosts are configured with a 1GB WT cache. It starts by inserting one million documents in batches of 100 with a 900 byte random string. When this completes (indicated by a blue vertical line in the image), the secondary of the test set is killed, preventing the primary from advancing its commit point or deleting old snapshots. It will create new snapshots until the limit of 1000 uncommitted snapshots is hit. After the secondary is killed, the workload switches to updating documents for 20 minutes. The updates are done in batches of 1000 sequential documents.&lt;/p&gt;

&lt;p&gt;The test set appears to use an unbounded amount of disk space and suffers from some extreme pauses. During some, but not all, of these pauses, the system seems to be completely idle with barely any CPU or disk utilization.&lt;/p&gt;

&lt;p&gt;To confirm that the problem was not related to there being 1000 snapshots I limited the server to keeping 3 total snapshots by setting the uncommitted snapshot limit to 2 at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.3.3/src/mongo/db/repl/oplog.cpp#L1100&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.3.3/src/mongo/db/repl/oplog.cpp#L1100&lt;/a&gt;. This didn&apos;t seem to make much of a difference.&lt;/p&gt;

&lt;p&gt;Also, moving the testSet.stop() line to above begineState(&apos;insert&apos;) will make the snapshots be of a empty collection, and all inserts will be after the snapshots. Even in this case, the disk usage seems to be unbounded.&lt;/p&gt;

&lt;p&gt;Repro:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Download the .js and .py files to a directory that contains a mongod binary&lt;/li&gt;
	&lt;li&gt;If needed, install the python2 libs pymongo and matplotlib&lt;/li&gt;
	&lt;li&gt;Launch a mongod on the default port (27017) for reporting and IPC&lt;/li&gt;
	&lt;li&gt;Run mongo workload.js (This will launch the replica sets, run monitor.py, and do the workload)&lt;/li&gt;
	&lt;li&gt;Once the workload starts run python plot.py (It will update as new data is collected)&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="275192">SERVER-23359</key>
            <summary>WiredTiger should not cache updates between named snapshots</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="mathias@mongodb.com">Mathias Stearn</reporter>
                        <labels>
                    </labels>
                <created>Fri, 25 Mar 2016 21:57:21 +0000</created>
                <updated>Tue, 6 Dec 2022 04:29:42 +0000</updated>
                            <resolved>Wed, 21 Dec 2016 23:30:28 +0000</resolved>
                                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="1463669" author="alexander.gorrod" created="Wed, 21 Dec 2016 23:30:09 +0000"  >&lt;p&gt;Cleaning up intermediate updates is incompatible with the requirements of future multi-document transaction support.&lt;/p&gt;</comment>
                            <comment id="1220822" author="michael.cahill" created="Thu, 31 Mar 2016 04:09:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=redbeard0531&quot; class=&quot;user-hover&quot; rel=&quot;redbeard0531&quot;&gt;redbeard0531&lt;/a&gt;, we did discuss discarding intermediate versions, that didn&apos;t make it into 3.2.  It shouldn&apos;t be too hard, but we need some additional tracking of transaction snapshots over what we maintain today.  I&apos;ll use this test case to measure how effective the solution is.&lt;/p&gt;</comment>
                            <comment id="1218259" author="redbeard0531" created="Tue, 29 Mar 2016 15:53:55 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt; When we were working on the design for this, it sounded like WT would automatically purge unneeded intermediate versions of documents. So if there is a snapshot at version 1 of a document, then it is updated 1000 times, it will only keep version 1 and the latest version. If this isn&apos;t the case, we may need to rethink our snapshot retention policy.&lt;/p&gt;</comment>
                            <comment id="1217526" author="michael.cahill" created="Tue, 29 Mar 2016 02:09:59 +0000"  >&lt;p&gt;This looks like expected behavior given the current design: WiredTiger keeps all updates after the oldest snapshot.  Once the cache becomes full, they overflow into the &quot;lookaside table&quot; (&lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt;).&lt;/p&gt;

&lt;p&gt;The next step should be to run the workload and gather diagnostic data to confirm that versions are overflowing into the lookaside table.  We should also confirm where threads are blocked when no progress is being made. &lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="115131" name="monitor.py" size="2624" author="mathias@mongodb.com" created="Fri, 25 Mar 2016 21:57:21 +0000"/>
                            <attachment id="115129" name="named_snapshot_behavior.png" size="321532" author="mathias@mongodb.com" created="Fri, 25 Mar 2016 21:57:21 +0000"/>
                            <attachment id="115132" name="plot.py" size="2224" author="mathias@mongodb.com" created="Fri, 25 Mar 2016 21:57:21 +0000"/>
                            <attachment id="115130" name="workload.js" size="4267" author="mathias@mongodb.com" created="Fri, 25 Mar 2016 21:57:21 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 29 Mar 2016 02:09:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 8 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 8 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkcev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrcylj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsf3u7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>