<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 02:57:54 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-1745] Query performance very poor when using $all</title>
                <link>https://jira.mongodb.org/browse/SERVER-1745</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;At random (seemingly) the performance of all of our queries using $all (and potentially others) have dropped significantly.  Queries that were taking 4 to 20 milliseconds are now taking anywhere from 4 to 10 seconds (sometimes even longer).  mongostat only shows a blip on the radar for the queries however (ie: it happens in under a second according to mongostat).  Here&apos;s an example query:&lt;/p&gt;

&lt;p&gt;----- code ---------&lt;br/&gt;
db.products.count({ &quot;variations.attributeIds&quot; : &lt;/p&gt;
{ &quot;$all&quot; : [ 3873 , 3898]}
&lt;p&gt;, &quot;enabled&quot; : true});&lt;br/&gt;
----- / code ---------&lt;/p&gt;

&lt;p&gt;another convenient query for timing is:&lt;br/&gt;
----- code -----&lt;br/&gt;
function timeThis() {&lt;br/&gt;
    var ret = new Date()+&quot;\n&quot;;&lt;br/&gt;
    ret &lt;ins&gt;= db.products.count({ &quot;variations.attributeIds&quot; : { &quot;$all&quot; : [ 3873 , 3898]}})&lt;/ins&gt;&quot;\n&quot;;&lt;br/&gt;
    ret += new Date();&lt;br/&gt;
    return ret;&lt;br/&gt;
}&lt;br/&gt;
timeThis();&lt;br/&gt;
----- /code -----&lt;/p&gt;

&lt;p&gt;That query takes anywhere from 4 to 8 seconds to execute.  CPU usage by the mongod process raises significantly (I&apos;m not sure if this is a bad thing).  I&apos;ve copied the dataset to a machine with 70GB of RAM (yes, 70 gigabytes) and 8 processor cores just to eliminate any potential hardware issues and the problems sill persist.&lt;/p&gt;

&lt;p&gt;the following indexes exist:&lt;br/&gt;
----- code ---------&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;uid&quot;:1}
&lt;p&gt;, &lt;/p&gt;
{unique:true, dropDups : true}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{weight:-1}
&lt;p&gt;);&lt;/p&gt;

&lt;p&gt;db.products.ensureIndex(&lt;/p&gt;
{&quot;dateCreated&quot;: 1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;dateCreated&quot;: -1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;/p&gt;

&lt;p&gt;db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.price&quot;: 1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.price&quot;: -1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;/p&gt;

&lt;p&gt;db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.salePercent&quot;: 1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.salePercent&quot;: -1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;/p&gt;

&lt;p&gt;db.products.ensureIndex(&lt;/p&gt;
{&quot;keywords&quot;: 1, &quot;weight&quot;: -1}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.attributeIds&quot;:1}
&lt;p&gt;);&lt;br/&gt;
db.products.ensureIndex(&lt;/p&gt;
{&quot;variations.attributeIds&quot;:1, &quot;enabled&quot;:1}
&lt;p&gt;);&lt;br/&gt;
----- / code ---------&lt;/p&gt;

&lt;p&gt;The dataset attached is half of the collection that is in production that is causing issues, i did the following to reduce it to half:&lt;/p&gt;

&lt;p&gt;----- code ---------&lt;br/&gt;
function reduceByHalf(colName) {&lt;br/&gt;
    var i = 0;&lt;br/&gt;
    var c = 0;&lt;br/&gt;
    db&lt;span class=&quot;error&quot;&gt;&amp;#91;colName&amp;#93;&lt;/span&gt;.find().forEach(function(doc) {&lt;br/&gt;
        if ((i%2)==0) {&lt;br/&gt;
            db&lt;span class=&quot;error&quot;&gt;&amp;#91;colName&amp;#93;&lt;/span&gt;.remove({_id:doc._id});&lt;br/&gt;
            c++;&lt;br/&gt;
        }&lt;br/&gt;
        i++;&lt;br/&gt;
    });&lt;br/&gt;
    return &quot;Deleted &quot;&lt;ins&gt;c&lt;/ins&gt;&quot; documents&quot;;&lt;br/&gt;
}&lt;br/&gt;
reduceByHalf(&quot;products&quot;);&lt;br/&gt;
----- / code ---------&lt;/p&gt;

&lt;p&gt;The dataset has roughly 16k documents in it and is attached to this issue in tar/gz format&lt;/p&gt;</description>
                <environment>Ubuntu lucid server with mongodb-stable package from 10gen&amp;#39;s apt repository</environment>
        <key id="13022">SERVER-1745</key>
            <summary>Query performance very poor when using $all</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="aaron">Aaron Staple</assignee>
                                    <reporter username="briandilley">Brian C. Dilley</reporter>
                        <labels>
                    </labels>
                <created>Mon, 6 Sep 2010 22:07:26 +0000</created>
                <updated>Mon, 16 Apr 2012 18:03:37 +0000</updated>
                            <resolved>Tue, 8 Feb 2011 18:15:30 +0000</resolved>
                                    <version>1.6.2</version>
                                                    <component>Performance</component>
                                        <votes>3</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="110336" author="ian@10gen.com" created="Mon, 16 Apr 2012 18:03:37 +0000"  >&lt;p&gt;@brian that is ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1748&quot; title=&quot;$all matching is slow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1748&quot;&gt;&lt;del&gt;SERVER-1748&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="110332" author="briandilley" created="Mon, 16 Apr 2012 17:50:28 +0000"  >&lt;p&gt;Where is the jira for the slow $all queries?&lt;/p&gt;</comment>
                            <comment id="23388" author="aaron" created="Tue, 8 Feb 2011 18:15:30 +0000"  >&lt;p&gt;Already filed a jira for the slow $all queries&lt;/p&gt;</comment>
                            <comment id="17853" author="aaron" created="Tue, 7 Sep 2010 20:38:36 +0000"  >&lt;p&gt;It looks like mongostat just shows the number of new operations per second, rather than the duration of any particular operation.  If you&apos;re interested in the time taken by a particular operation, you can use the http interface &amp;lt;&lt;a href=&quot;http://www.mongodb.org/display/DOCS/Http+Interface&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.mongodb.org/display/DOCS/Http+Interface&lt;/a&gt;&amp;gt; or look at the mongod log.&lt;/p&gt;</comment>
                            <comment id="17851" author="briandilley" created="Tue, 7 Sep 2010 20:14:18 +0000"  >&lt;p&gt;The other thing that is worth mentioning is that mongostat showed the queries completing in (or under) a second when they obviously did not.&lt;/p&gt;</comment>
                            <comment id="17848" author="aaron" created="Tue, 7 Sep 2010 19:04:14 +0000"  >&lt;p&gt;Thanks for reporting this - you&apos;re right, it&apos;s slow.  I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1748&quot; title=&quot;$all matching is slow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1748&quot;&gt;&lt;del&gt;SERVER-1748&lt;/del&gt;&lt;/a&gt; to represent a fix for the slow component.&lt;/p&gt;</comment>
                            <comment id="17828" author="briandilley" created="Tue, 7 Sep 2010 02:32:45 +0000"  >&lt;p&gt;So i continued to play with this and found that sometimes variations.attributeIds contained over 3K entries (many times actually) and that&apos;s what was slowing things down.  Code elsewhere was doing this and causing that to happen.  So if you want to close this bug on those grounds that&apos;s fine - unless you think it should still perform.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="13029">SERVER-1748</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="10429" name="dump.tar.gz" size="69362499" author="briandilley" created="Mon, 6 Sep 2010 22:07:26 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 7 Sep 2010 19:04:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 44 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-1748'>SERVER-1748</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 44 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aaron</customfieldvalue>
            <customfieldvalue>briandilley</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrpfrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrij53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21753</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs52mn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>