Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-1779

"finding median" too slow.

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.7.1
    • Affects Version/s: 1.6.2, 1.7.0
    • Component/s: Performance
    • Labels:
      None
    • Environment:
      ubuntu 9.10
      24G RAM

      the db is on a RAID0 of 2 disks
    • Linux

      Here's some excerpt from the logs
      Sun Sep 12 10:51:14 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 24430203 } and { : 25662603 } took 3161 ms.
      Sun Sep 12 10:51:36 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 26814034 } and { : 27937206 } took 14448 ms.
      Sun Sep 12 10:52:10 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 22297203 } and { : 23324407 } took 2012 ms.
      Sun Sep 12 10:52:55 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 60706232 } and { : 62958603 } took 23454 ms.
      Sun Sep 12 10:53:05 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 25662603 } and { : 26814034 } took 3050 ms.
      Sun Sep 12 10:53:09 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 23324407 } and { : 24430203 } took 2804 ms.
      Sun Sep 12 10:53:34 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 37270223 } and { : 38933407 } took 24029 ms.
      Sun Sep 12 10:54:19 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 40766434 } and { : 42625800 } took 13718 ms.
      Sun Sep 12 10:54:53 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 58696810 } and { : 60706232 } took 11311 ms.
      Sun Sep 12 10:55:11 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 30362400 } and { : 31572033 } took 9520 ms.
      Sun Sep 12 10:55:35 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 72665435 } and { : 96292200 } took 7396 ms.
      Sun Sep 12 10:55:53 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 68727613 } and { : 72665435 } took 7880 ms.
      Sun Sep 12 10:56:04 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 38933407 } and { : 40766434 } took 10045 ms.
      Sun Sep 12 10:56:43 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 35630403 } and { : 37270223 } took 7486 ms.
      Sun Sep 12 14:31:02 [conn10] Finding median for index:

      { uiq: 1.0 }

      between { : 65599205 } and { : 68727613 } took 95402 ms.
      Sun Sep 12 14:32:34 [conn10] Finding median for index:

      { uiq: 1.0 }

      between { : 62958603 } and { : 65599205 } took 72608 ms.
      Sun Sep 12 14:34:19 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 56850022 } and { : 58696810 } took 58778 ms.
      Sun Sep 12 14:35:20 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 55276200 } and { : 56850022 } took 50957 ms.
      Sun Sep 12 14:43:17 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 45860418 } and { : 47944851 } took 47859 ms.
      Sun Sep 12 14:44:10 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 51170454 } and { : 52417803 } took 38282 ms.
      Sun Sep 12 14:44:47 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 53762457 } and { : 55276200 } took 27610 ms.
      Sun Sep 12 14:45:55 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 42625800 } and { : 44283041 } took 34589 ms.
      Sun Sep 12 14:46:19 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 47944851 } and { : 49876203 } took 17147 ms.
      Sun Sep 12 14:47:07 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 49876203 } and { : 51170454 } took 23866 ms.
      Sun Sep 12 14:47:48 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 44283041 } and { : 45860418 } took 18224 ms.
      Sun Sep 12 14:48:53 [conn6] Finding median for index:

      { uiq: 1.0 }

      between { : 52417803 } and { : 53762457 } took 21154 ms.

      The indexes surely fit in memory. The whole DB does!

      > db.answers.stats()
      {
      "sharded" : true,
      "ns" : "pravdorub_production.answers",
      "count" : 273039209,
      "size" : 33919877756,
      "avgObjSize" : 124.23079410547223,
      "storageSize" : 39745093120,
      "nindexes" : 2,
      "nchunks" : 92,
      "shards" : {
      "brussels" : {
      "ns" : "pravdorub_production.answers",
      "count" : 136990615,
      "size" : 17019609560,
      "avgObjSize" : 124.23923755652896,
      "storageSize" : 19872546560,
      "numExtents" : 40,
      "nindexes" : 2,
      "lastExtentSize" : 1991168256,
      "paddingFactor" : 1,
      "flags" : 1,
      "totalIndexSize" : 14175980928,
      "indexSizes" :

      { "_id_" : 6682338992, "uiq_1" : 7493641936 }

      ,
      "ok" : 1
      },
      "moscow" : {
      "ns" : "pravdorub_production.answers",
      "count" : 136048594,
      "size" : 16900268196,
      "avgObjSize" : 124.22229219068592,
      "storageSize" : 19872546560,
      "numExtents" : 40,
      "nindexes" : 2,
      "lastExtentSize" : 1991168256,
      "paddingFactor" : 1,
      "flags" : 1,
      "totalIndexSize" : 13670329728,
      "indexSizes" :

      { "_id_" : 6047942320, "uiq_1" : 7622387408 }

      ,
      "ok" : 1
      }
      },
      "ok" : 1
      }

      > db.stats()
      {
      "raw" : {
      "brussels/moskau8:27018" :

      { "collections" : 3, "objects" : 137156015, "avgObjSize" : 124.23939636916398, "dataSize" : 17040180512, "storageSize" : 19872554752, "numExtents" : 42, "indexes" : 2, "indexSize" : 14192291200, "fileSize" : 45009076224, "ok" : 1 }

      ,
      "moscow/moskau7:27018" :

      { "collections" : 3, "objects" : 136252914, "avgObjSize" : 124.22246986952514, "dataSize" : 16925673504, "storageSize" : 19872554496, "numExtents" : 42, "indexes" : 2, "indexSize" : 13691178368, "fileSize" : 45009076224, "ok" : 1 }

      },
      "objects" : 273408929,
      "avgObjSize" : 124.23096107442782,
      "dataSize" : 33965854016,
      "storageSize" : 39745109248,
      "numExtents" : 84,
      "indexes" : 4,
      "indexSize" : 27883469568,
      "fileSize" : 90018152448,
      "ok" : 1
      }

            Assignee:
            eliot Eliot Horowitz (Inactive)
            Reporter:
            sergei.tulentsev@gmail.com Sergei Tulentsev
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: