[SERVER-12651] Unindexed distinct queries regression (wrt 2.4) Created: 07/Feb/14  Updated: 11/Jul/16  Resolved: 12/Feb/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.5.5
Fix Version/s: 2.6.0-rc0

Type: Bug Priority: Major - P3
Reporter: Davide Italiano Assignee: Benety Goh
Resolution: Done Votes: 0
Labels: 26qa, Performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File callgrind.distinct     File distinct.js    
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Unindexed distinct queries are slower in 2.6 WRT 2.4, and this is noticeable both in single-threaded and multi-threaded tests I ran. An easy (single-threaded) way to reproduce it can be found attached.

27fb434eaebc35646cdadf7acac562aa47c06ac8 (HEAD as per today) output:

connecting to: test
Trial number: 0 1608
Trial number: 1 1603
Trial number: 2 1678
Trial number: 3 1604
Trial number: 4 1597

2.4.9 output:

connecting to: test
Trial number: 0 1273
Trial number: 1 1272
Trial number: 2 1269
Trial number: 3 1271
Trial number: 4 1266



 Comments   
Comment by Githook User [ 12/Feb/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-12651 drop projection when collection scan is only solution for distinct command
Branch: master
https://github.com/mongodb/mongo/commit/8c99c9ad5cd5ef14f808c1c135f807255330f025

Comment by Davide Italiano [ 07/Feb/14 ]

Apparently the biggest offender is ProjectionExec::transform.
Digging into shows the following two hotspots:

     1      3  262:             BSONObjIterator it(_source);
     .      .  263:             while (it.more()) {
     8    147  264:                 BSONElement specElt = it.next();
   117    248  265:                 if (mongoutils::str::equals("_id", specElt.fieldName())) {
     .      .  266:                     continue;
     .      .  267:                 }

    10     21  269:                 BSONElement keyElt;
     .      .  270:                 // We can project a field that doesn't exist.  We just ignore it.
    20    469  271:                 if (member->getFieldDotted(specElt.fieldName(), &keyElt) && !keyElt.eoo()) {
     6     96  272:                     bob.appendAs(keyElt, specElt.fieldName());
     .      .  273:                 }

Comment by Davide Italiano [ 07/Feb/14 ]

Attach profiling output + pprof (top 5) output:

     645  10.6%  10.6%      645  10.6% __nss_hosts_lookup
     307   5.1%  15.7%      310   5.1% __libc_send
     166   2.7%  18.4%      244   4.0% mongo::compareElementValues
     150   2.5%  20.9%     1168  19.3% mongo::ProjectionExec::transform@9531a0
     108   1.8%  22.7%      108   1.8% __libc_recv
     107   1.8%  24.5%      649  10.7% mongo::getFieldDottedOrArray
      97   1.6%  26.1%       97   1.6% 0x00007fffb33fedff
      93   1.5%  27.6%       93   1.5% mongo::canonicalizeBSONType
      88   1.5%  29.1%      110   1.8% mongo::BSONElementIterator::next
      87   1.4%  30.5%      204   3.4% mongo::BSONElement::size

Generated at Thu Feb 08 03:29:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.