<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:36:56 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34518] WT Transaction / Timestamp error during non-responsiveness</title>
                <link>https://jira.mongodb.org/browse/SERVER-34518</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We recently migrated one of our replicasets to 3.6.3 from 3.4. It had been running just fine for about 4 days when we had a rash of slow queries out of nowhere, the database became super-slow to unresponsive. During that time, the only indication of problems other than log data showing slow queries were the following messages (by the ton):&lt;/p&gt;

&lt;p&gt;	Line 181225: 2018-04-16T07:28:10.035-0500 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn908492&amp;#93;&lt;/span&gt; WiredTiger error (22) &lt;span class=&quot;error&quot;&gt;&amp;#91;1523881690:35506&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;8828:0x7fbc77746700&amp;#93;&lt;/span&gt;, WT_SESSION.begin_transaction: read timestamp 5ad496da00000001 older than oldest timestamp 5ad496da00000004: Invalid argument&lt;/p&gt;</description>
                <environment></environment>
        <key id="529389">SERVER-34518</key>
            <summary>WT Transaction / Timestamp error during non-responsiveness</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="sallgeud">Chad Kreimendahl</reporter>
                        <labels>
                    </labels>
                <created>Tue, 17 Apr 2018 15:32:43 +0000</created>
                <updated>Wed, 16 Oct 2019 13:31:16 +0000</updated>
                            <resolved>Tue, 24 Jul 2018 17:35:41 +0000</resolved>
                                    <version>3.6.3</version>
                                                    <component>Replication</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1953155" author="dmitry.agranat" created="Sun, 22 Jul 2018 11:04:20 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;As mentioned earlier, after carefully reviewing the last set of uploaded data, we were not able to find any issue with the originally reported &quot;WT Transaction / Timestamp error during non-responsiveness&quot;. As such, we will close this case. If you still experience issues with the &quot;WT Transaction / Timestamp error during non-responsiveness&quot;, please upload the archived &lt;tt&gt;mongoD&lt;/tt&gt; log, &lt;tt&gt;diagnostic.data&lt;/tt&gt;, the exact time of the issue and we will reopen this case.&lt;/p&gt;

&lt;p&gt;As for the mentioned issue with how MongoDB clients route read operations, if you suspect this is a bug, please open a separate case and we will be happy to investigate it.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="1946887" author="sallgeud" created="Fri, 13 Jul 2018 16:25:37 +0000"  >&lt;p&gt;Every single query that shows&#160;&quot;secondaryPreferred&quot; is wrong. 100% of those queries are at least 2-20x slower than they are in version 3.4.&#160; They are also 100% &lt;b&gt;NOT&lt;/b&gt; secondary queries. This is entirely a primary system. Not a single line of our code EVER queries from a secondary. So we should never, ever, ever see&#160;&quot;secondaryPreferred&quot;.&lt;/p&gt;</comment>
                            <comment id="1942702" author="dmitry.agranat" created="Tue, 10 Jul 2018 05:23:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;After carefully reviewing the last set of logs and diagnostic data, we&#8217;re not able to find the necessary clues/log lines that you mention, did we miss anything? Can you please re-check on your end and point us to a timestamp/logline where this issue occurred?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;
</comment>
                            <comment id="1939750" author="sallgeud" created="Thu, 5 Jul 2018 18:21:05 +0000"  >&lt;p&gt;Ramon,&lt;/p&gt;

&lt;p&gt;I&apos;ve uploaded an up-to-date set of both new diagnostic-data directory (in a tar file) and log files from the past 4 days.&#160; My hope is that this covers you for what you need.&lt;/p&gt;</comment>
                            <comment id="1924065" author="ramon.fernandez" created="Mon, 18 Jun 2018 20:00:18 +0000"  >&lt;p&gt;Yes &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;, the &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/6d3ef103-9fa1-4e6a-9efe-603c3aa1cebf.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure portal&lt;/a&gt; for this ticket is still available.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1923868" author="sallgeud" created="Mon, 18 Jun 2018 18:18:02 +0000"  >&lt;p&gt;We&apos;ll have to just do the logs and the diagnostic data uploads at the same time. Will the same URL work?&lt;/p&gt;</comment>
                            <comment id="1921024" author="dmitry.agranat" created="Thu, 14 Jun 2018 19:05:59 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt; for uploading the archived content of &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt;. Could you also upload the covering &lt;tt&gt;mongod&lt;/tt&gt; log and provide the exact time (including the time zone) of non-responsiveness?&lt;/p&gt;</comment>
                            <comment id="1919967" author="sallgeud" created="Wed, 13 Jun 2018 19:11:04 +0000"  >&lt;p&gt;We&apos;re definitely still seeing the error. I&apos;ve started an upload of a tar of that directory now. It&apos;s quite large, but will be done in the next hour.&lt;/p&gt;</comment>
                            <comment id="1918888" author="dmitry.agranat" created="Tue, 12 Jun 2018 20:27:26 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The log message that you are seeing is by design. During periods of slowness, it is possible for the oldest timestamp to race ahead of the oplog read timestamp. When this error occurs, the oplog read simply retries the operation using an updated read timestamp. &lt;/p&gt;

&lt;p&gt;To understand the reason of the slowness, we&apos;ve requested archived content of &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; which I could not find under the &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/6d3ef103-9fa1-4e6a-9efe-603c3aa1cebf.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure portal&lt;/a&gt; we&apos;ve provided. Since the reported error occurred on March 16th, I suspect the content of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; was already overwritten. If this issue is still a problem for you, please provide an up to date:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;archived content of $dbpath/diagnostic.data&lt;/li&gt;
	&lt;li&gt;complete mongod log files ideally covering time from upgrade until the present&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;



</comment>
                            <comment id="1913127" author="sallgeud" created="Wed, 6 Jun 2018 22:16:51 +0000"  >&lt;p&gt;On that note:&#160; The only queries that run on the secondaries are things that are run manually.&#160; Yet, nearly every single slow query (over 95%+ at last check) showed that it was being queried as &quot;secondaryPreferred&quot;, even though the system logging it &lt;b&gt;is&lt;/b&gt;&#160;the primary, and every single one of the queries is set to be primary.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The slowness is also frequent, in that it happens at least several times an hour, between 10 and 100 times per hour. It happens only in short bursts, such that you&apos;ll get a rash of slow queries all within a few seconds... then all is well for a long period of time (minutes).&lt;/p&gt;</comment>
                            <comment id="1912577" author="bgentry" created="Wed, 6 Jun 2018 16:20:22 +0000"  >&lt;p&gt;Chad asked me to upload the log files to the private portal, which I have done.&#160; After the 3.6 node became primary, you will also see a lot of slow (multi-second) queries with&#160;$readPreference: { mode: &quot;secondaryPreferred&quot; }.&#160; We are specifying primaryPreferred (via the C# driver), so we don&apos;t understand why secondaryPreferred is being used.&#160; They seem to occur in batches every minute or so.&#160; This did not occur when the 3.4 node was primary.&lt;/p&gt;</comment>
                            <comment id="1866773" author="bruce.lucas@10gen.com" created="Tue, 17 Apr 2018 18:57:25 +0000"  >&lt;p&gt;Hi Chad,&lt;/p&gt;

&lt;p&gt;So we can investigate it can you please upload to &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/6d3ef103-9fa1-4e6a-9efe-603c3aa1cebf.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this secure private portal&lt;/a&gt;, for the affected instance:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;archived content of $dbpath/diagnostic.data&lt;/li&gt;
	&lt;li&gt;complete mongod log files ideally covering time from upgrade until the present&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also can you please give me a timeline of the events that you describe as accurately as you can (including timezone). This will ensure that we&apos;re looking at the right time period in the data.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000dOs12QAC, 5002K00000hRAN9QAO]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Apr 2018 18:57:25 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 29 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 29 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bgentry</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htvmtj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htmsev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htv91r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>