<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:59:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42013] Provide table counts (collections + indexes) like db.stats() but for the instance level</title>
                <link>https://jira.mongodb.org/browse/SERVER-42013</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;At the moment the only known efficient way to get a system-wide table count for a mongod instance is to walk each of the databases and each of the collections getting a count of the total items.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Why this matters:&lt;/p&gt;

&lt;p&gt;Our scalability issues, because of our structure, are entirely based on number of files. As we approach ~350,000 collections + indexes, we begin to experience numerous issues. So we run multiple instances per system of mongo, split on various ports, in order to keep that number down, yet properly utilize our Optane drives and memory.&lt;/p&gt;

&lt;p&gt;Our solution to tracking this information systematically has been to have a tool that connects to each database, walks each db and counts the tables. We then report this back into a metrics database. We prioritize the creation of new dbs and new instances of mongo based on how many files.&#160; We&apos;ve found the best mid-line to be the ~100k file range.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Running the above query is somewhat expensive and causes unnecessary utilization, and may load unneeded data into memory. I believe wiredTiger, internally, tracks its open tables and would be able to report this number (even if only an estimate) in its statistics.&lt;/p&gt;</description>
                <environment></environment>
        <key id="828357">SERVER-42013</key>
            <summary>Provide table counts (collections + indexes) like db.stats() but for the instance level</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="12300">Won&apos;t Do</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="sallgeud">Chad Kreimendahl</reporter>
                        <labels>
                    </labels>
                <created>Fri, 28 Jun 2019 20:44:14 +0000</created>
                <updated>Tue, 6 Dec 2022 02:55:29 +0000</updated>
                            <resolved>Mon, 8 Jun 2020 18:39:30 +0000</resolved>
                                    <version>4.0.10</version>
                                                    <component>Diagnostics</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2373585" author="daniel.hatcher" created="Thu, 15 Aug 2019 15:17:22 +0000"  >&lt;p&gt;I will pass along the request to get the equivalent of the &lt;tt&gt;dbStats&lt;/tt&gt; command but for the whole &lt;tt&gt;mongod&lt;/tt&gt; instance.&lt;/p&gt;</comment>
                            <comment id="2363557" author="mongo@phish.org" created="Thu, 8 Aug 2019 20:08:18 +0000"  >&lt;p&gt;So really instead of having to iterate every single database and count the collections and indexes, would it be possible to get this stat aggregated in a more efficient way to the mongod level. such that one query would get it vs say 250+ (in our case)&lt;/p&gt;</comment>
                            <comment id="2346948" author="mongo@phish.org" created="Sun, 28 Jul 2019 03:18:38 +0000"  >&lt;p&gt;I think dbstats covers it. I&apos;m not sure why we spaced on seeing this. We&apos;re pulling dbstats on every database and creating trend information for them daily, but guess we were serializing and deserializing without including some fields that&apos;re in there.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;So that just means it&apos;s a bit inefficient to get the whole mongod count, in that you have to query every database for its stats. It would be wonderful if the system level stats would return this aggregated information in a more rapid form.&lt;/p&gt;</comment>
                            <comment id="2313712" author="daniel.hatcher" created="Fri, 5 Jul 2019 17:46:19 +0000"  >&lt;p&gt;I apologize, I&apos;m still a little unclear as to your use-case. You can access both &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/dbStats/index.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;dbStats&lt;/a&gt; (which provides a count of all collections and indexes within a given database) and &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/serverStatus/index.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;serverStatus&lt;/a&gt; (which provides a count of all open tables on a given &lt;tt&gt;mongod&lt;/tt&gt;) through any of our drivers. It shouldn&apos;t be difficult to incorporate either method programatically. Are you trying to capture a statistic other than a count of all collections / indexes or a count of all currently open tables?&lt;/p&gt;</comment>
                            <comment id="2311924" author="sallgeud" created="Wed, 3 Jul 2019 23:34:45 +0000"  >&lt;p&gt;I guess for us we want to auto deploy the next customer onto the cluster with the least files automatically. A quick table count method would be the most efficient&lt;/p&gt;</comment>
                            <comment id="2311923" author="sallgeud" created="Wed, 3 Jul 2019 23:33:22 +0000"  >&lt;p&gt;Not really. Since we need a programatic way to get the info, we&apos;re stuck using the C# driver. So open handles helps a bit, but doesn&apos;t get us there. &lt;/p&gt;

&lt;p&gt;Counting files on the system is something we do, but wouldn&apos;t work, long term as we don&apos;t want to manage this with yet another process running on another set of hardware (mongo servers, Linux)&lt;/p&gt;</comment>
                            <comment id="2307513" author="daniel.hatcher" created="Mon, 1 Jul 2019 20:33:08 +0000"  >&lt;p&gt;Hello Chad,&lt;/p&gt;

&lt;p&gt;As WiredTiger is currently (as of 4.0.10) a file-per-collection and a file-per-index storage engine, performing a count of files in the data directory should give you an idea of the maximum number of tables your instance will open. If you&apos;re looking for tables that are currently held open, we keep track of &quot;open data-handles&quot; in our &lt;tt&gt;serverStatus&lt;/tt&gt; output under &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/serverStatus/index.html#wiredtiger&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&quot;wiredTiger&quot;.&quot;data-handle&quot;.&quot;connection data handles currently active&quot;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Are these statistics insufficient for your use case?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 1 Jul 2019 20:25:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 25 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 25 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>mongo@phish.org</customfieldvalue>
            <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv9ihj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr54mv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv94qv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>