<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:05:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44345] MongoS crash with &quot;BufBuilder attempted to grow()&quot; above 64MB while restarting/upgrading a secondary from 3.4 to 3.6</title>
                <link>https://jira.mongodb.org/browse/SERVER-44345</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We are in the middle of a 3.4.17-evg1 to 3.6.14 upgrade, when one of the mongos servers crashed.&lt;/p&gt;

&lt;p&gt;The most specific message line is this:&lt;/p&gt;

&lt;p&gt;Assertion: 13548:BufBuilder attempted to grow() to 1751919127 bytes, past the 64MB limit. src/mongo/bson/util/builder.h 326&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;It happened right after a log line where one of our nodes (in the trace attached I&apos;ve renamed it to &amp;lt;SHARD9_SECONDARY1&amp;gt;) was just starting to shut down as a part of the upgrade process.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;This happened less than an hour before I hit submit on this report, so if there are any transient logs our debug output you want me to provide, let me know!&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;FYI For 3.4.17-evg1, the &quot;-evg1&quot; is just our custom build patched version with 3 logging changes described in the description of this bug&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43021&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-43021&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Note that a few months ago, our 3.6 cluster (we have a different, less high-impact cluster we already have at 3.6) had an issue where something tried to write more than 16MB, and it crahsed &lt;em&gt;all&lt;/em&gt; of our mongoS servers in succession.&#160; That bug is herehttps://jira.mongodb.org/browse/SERVER-43021&#160;just in case it&apos;s helpful.&#160; We never resolved that bug, but we also never saw the issue again (luckily).&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="986775">SERVER-44345</key>
            <summary>MongoS crash with &quot;BufBuilder attempted to grow()&quot; above 64MB while restarting/upgrading a secondary from 3.4 to 3.6</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="glajchs">Scott Glajch</reporter>
                        <labels>
                    </labels>
                <created>Thu, 31 Oct 2019 20:49:03 +0000</created>
                <updated>Wed, 11 Dec 2019 22:05:24 +0000</updated>
                            <resolved>Tue, 5 Nov 2019 21:38:00 +0000</resolved>
                                    <version>3.4.17</version>
                    <version>3.6.14</version>
                                                    <component>Upgrade/Downgrade</component>
                                        <votes>1</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2599618" author="xgen-internal-githook" created="Wed, 11 Dec 2019 18:23:57 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Billy Donahue&apos;, &apos;email&apos;: &apos;billy.donahue@mongodb.com&apos;, &apos;username&apos;: &apos;BillyDonahue&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44345&quot; title=&quot;MongoS crash with &amp;quot;BufBuilder attempted to grow()&amp;quot; above 64MB while restarting/upgrading a secondary from 3.4 to 3.6&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44345&quot;&gt;&lt;del&gt;SERVER-44345&lt;/del&gt;&lt;/a&gt; stack trace all threads&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4f0b6507d4acd185b4c925c176a5693be339f364&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4f0b6507d4acd185b4c925c176a5693be339f364&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;(typo: that git commit should have said &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33445&quot; title=&quot;Add signal handler to generate stack traces&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33445&quot;&gt;&lt;del&gt;SERVER-33445&lt;/del&gt;&lt;/a&gt;)&lt;/p&gt;</comment>
                            <comment id="2518981" author="daniel.hatcher" created="Tue, 5 Nov 2019 21:37:48 +0000"  >&lt;p&gt;I believe this is being worked on in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43021&quot; title=&quot;MongoS server crashes when attempt to update single record &amp;gt; 16MB (16793648)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43021&quot;&gt;&lt;del&gt;SERVER-43021&lt;/del&gt;&lt;/a&gt; so I&apos;m going to close this ticket as a duplicate. Please let me know if this is actually a different issue. &lt;/p&gt;</comment>
                            <comment id="2514625" author="glajchs" created="Fri, 1 Nov 2019 19:52:47 +0000"  >&lt;p&gt;This happened again today, with the same looking stack trace.&#160; We are still in the process of restarting our mongod nodes and again it coincided with one of them coming down for upgrade.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="905193">SERVER-43021</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="235876" name="shard9_secondary1_crash_10_31_2019.txt" size="8932" author="glajchs" created="Thu, 31 Oct 2019 20:46:25 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 5 Nov 2019 21:37:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 9 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>xgen-internal-githook</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 9 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>glajchs</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw02v3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvofrb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;(assuming the upgrade was related)&lt;br/&gt;
1.&#160; Have a 3.4 cluster&lt;/p&gt;

&lt;p&gt;2. Upgrade the config servers&lt;/p&gt;

&lt;p&gt;3. Start upgrading the replica sets&lt;/p&gt;

&lt;p&gt;4. Eventually one of them secondary restarts on the replica sets triggers this exception.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvzp4f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>