<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:25:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31063] Monitor queued network data for adaptive service executor</title>
                <link>https://jira.mongodb.org/browse/SERVER-31063</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The adaptive service executor introduces a significant new queueing point. Two runs of test that ramps up to 20 k connections executing updates - first run is adaptive service executor, second run is synchronous (default) executor.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/165329/165329_queueing.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;In the sync case (right) the signficant queuing point is the &quot;queued writers&quot; metric, but not in the adaptive case.&lt;/p&gt;

&lt;p&gt;In the adaptive case (left) queuing occurs upstream of that. By monitoring /proc/net/tcp (&quot;csv rx&quot; chart) we can see far more requests queued in the network layer in the adaptive case than in the sync case. This represents about 1800 queued requests, so is a very significant queuing point, unlike in the sync case. Note that this only includes network data buffered in the kernel; there may possibly be signficant additional network data buffered in ASIO.&lt;/p&gt;

&lt;p&gt;Understanding the queueing points is important for diagnosing bottleneck, so we should monitor buffered network data to support the adaptive service executor. This could be done&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;at the kernel level (as above). However this would be global for the node (not specific to mongod) and would not include data buffered in ASIO.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;at the socket layer in mongod. This would be specific to mongod, but would not include whatever might be buffered in ASIO.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;via ASIO APIs, if available.&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment>&lt;br/&gt;
</environment>
        <key id="426682">SERVER-31063</key>
            <summary>Monitor queued network data for adaptive service executor</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="12300">Won&apos;t Do</resolution>
                                        <assignee username="backlog-server-servicearch">Backlog - Service Architecture</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>SWDI</label>
                            <label>re-triaged-ticket</label>
                    </labels>
                <created>Tue, 12 Sep 2017 18:36:45 +0000</created>
                <updated>Mon, 8 Jan 2024 15:22:52 +0000</updated>
                            <resolved>Tue, 2 Nov 2021 15:23:34 +0000</resolved>
                                                                    <component>Diagnostics</component>
                                        <votes>24</votes>
                                    <watches>27</watches>
                                                                                                                <comments>
                            <comment id="4163407" author="JIRAUSER1262719" created="Tue, 2 Nov 2021 15:23:34 +0000"  >&lt;p&gt;The Service Arch team is in the process of cleaning up tickets in the backlog. This ticket has not been updated in two years so we are closing it. Please reopen if you think this change is valuable.&lt;/p&gt;</comment>
                            <comment id="1754890" author="bruce.lucas@10gen.com" created="Mon, 18 Dec 2017 19:52:26 +0000"  >&lt;p&gt;It looks like you&apos;re correct about SIOCGSTAMP.&lt;/p&gt;

&lt;p&gt;There is an an &lt;a href=&quot;https://www.kernel.org/doc/Documentation/networking/timestamping.txt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SO_TIMESTAMP ioctl&lt;/a&gt; that purports to be able to provide timestamps for stream-based network i/o like TCP, but I don&apos;t know it&apos;s status; it may require a sufficiently recent kernel.&lt;/p&gt;

&lt;p&gt;I agree that we should record /proc/net/tcp, specifically tx_queue and rx_queue, in FTDC. That will at least give us queue sizes.&lt;/p&gt;</comment>
                            <comment id="1750206" author="henrik.edin" created="Tue, 12 Dec 2017 21:10:19 +0000"  >&lt;p&gt;From what I understand you can only check SIOCGSTAMP for datagram sockets. It probably doesn&apos;t make much sense on tcp the way the protocol works.&lt;br/&gt;
ASIO doesn&apos;t do any additional buffering. What it looks like to me is that this mainly is an artifact of using asynchronous networking and not the queuing before a worker thread picks up the task in the adaptive executor.&lt;br/&gt;
As I wrote in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30623&quot; title=&quot;Operation latency and queuing stats are incorrect for adaptive executor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30623&quot;&gt;&lt;del&gt;SERVER-30623&lt;/del&gt;&lt;/a&gt; I think it&apos;s a good idea to add /proc/net/tcp to FTDC. But I&apos;m not sure if there&apos;s anything else we should do. Do you have any opinion on this &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jonathan.reams&quot; class=&quot;user-hover&quot; rel=&quot;jonathan.reams&quot;&gt;jonathan.reams&lt;/a&gt;?&lt;/p&gt;</comment>
                            <comment id="1748533" author="bruce.lucas@10gen.com" created="Mon, 11 Dec 2017 16:51:16 +0000"  >&lt;p&gt;We could also investigate whether the SIOCGSTAMP ioctl would provide the means to monitor time spent queued in the kernel network buffers so that it an be added to the reported latency (along with other additions to latency reporting in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30623&quot; title=&quot;Operation latency and queuing stats are incorrect for adaptive executor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30623&quot;&gt;&lt;del&gt;SERVER-30623&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;</comment>
                            <comment id="1670948" author="bruce.lucas@10gen.com" created="Tue, 12 Sep 2017 18:38:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jonathan.reams&quot; class=&quot;user-hover&quot; rel=&quot;jonathan.reams&quot;&gt;jonathan.reams&lt;/a&gt;, I was a bit surprised to see requests queueing in the network (&quot;csv rx&quot;) instead of in the task queue (&quot;totalTimeQueuedMicros&quot;) here, so maybe there&apos;s also an issue in the code to be investigated here.&lt;/p&gt;

&lt;p&gt;(Note &quot;totalTimeQueuedMicros&quot; is marked with unit of &quot;threads&quot; which is not the correct unit in the case of the adaptive executor - should be &quot;tasks&quot;).&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="415602">SERVER-30623</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="165329" name="queueing.png" size="133764" author="bruce.lucas@mongodb.com" created="Tue, 12 Sep 2017 18:36:44 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25132"><![CDATA[Service Arch]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 Dec 2017 21:10:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 14 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1120</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 14 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-servicearch</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>henrik.edin@mongodb.com</customfieldvalue>
            <customfieldvalue>lauren.lewis@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjre7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8iun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hteaif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>