Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9078

MongoDB2.4 cluster insert much more slow than 2.2.

    XMLWordPrintable

Details

    • Question
    • Status: Closed
    • Major - P3
    • Resolution: Cannot Reproduce
    • 2.4.0
    • None
    • Performance
    • None
    • Linux hz166-97 2.6.38-2-amd64
      Debian6.0
      2 MongoDB test cluster. each cluster composed of 3 shards.
      One cluster deployed MongoDB2.4 and the other 2.2.3.

    Description

      Hi,
      I installed MongoDB2.4 recently and found that the insert performance is much more
      pool than it was in 2.2.3.

      Please help to try to find the reason.

      Test schema:
      Load 330W records from a mysql table and insert them into mongodb cluster, mark down
      the insert time.
      2.4 cluster total insert time: 630 seconds
      2.2 cluster total insert time: 110 seconds

      For higher insert speed, I pre split the collection into 60 chunks, each node contains
      20 chunks. The shard key is (second of timestamp) << 32 + timestamp.

      And I found that the io load of 2.4 is higher than 2.2:

      iostax -m result for the test period(of one node):
      2.4 cluster:

      Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
      sda             112.22         0.00         5.35          0       3371

      2.2 cluster:

      Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
      sda              75.69         0.00         9.55          0       2386

      2.4 collection stats:

      mongos> db.gm_item.stats()
      {
              "sharded" : true,
              "ns" : "tw2_log_10148.gm_item",
              "count" : 3252213,
              "numExtents" : 48,
              "size" : 1907352352,
              "storageSize" : 2203701248,
              "totalIndexSize" : 974219456,
              "indexSizes" : {
                      "_id_" : 105543984,
                      "_sk_1" : 160118784,
                      "dst_item_uuid_ex_1" : 82315968,
                      "dst_role_1" : 150765440,
                      "op_nuid_1" : 122656352,
                      "src_item_uuid_ex_1" : 75987744,
                      "src_role_1" : 186036704,
                      "t_when_1" : 90794480
              },
              "avgObjSize" : 586.4783001605368,
              "nindexes" : 8,
              "nchunks" : 60,
              "shards" : {
                      "mds1" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1085111,
                              "size" : 636530928,
                              "avgObjSize" : 586.6044377026866,
                              "storageSize" : 786976768,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 210907136,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 331463216,
                              "indexSizes" : {
                                      "_id_" : 35214032,
                                      "_sk_1" : 53389280,
                                      "t_when_1" : 30292080,
                                      "op_nuid_1" : 40863648,
                                      "src_role_1" : 62121248,
                                      "src_item_uuid_ex_1" : 25370128,
                                      "dst_role_1" : 56864080,
                                      "dst_item_uuid_ex_1" : 27348720
                              },
                              "ok" : 1
                      },
                      "mds2" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1081248,
                              "size" : 633920176,
                              "avgObjSize" : 586.2856402971381,
                              "storageSize" : 708362240,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 189837312,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 310622592,
                              "indexSizes" : {
                                      "_id_" : 35091392,
                                      "_sk_1" : 53242112,
                                      "t_when_1" : 30185792,
                                      "op_nuid_1" : 40814592,
                                      "src_role_1" : 61826912,
                                      "src_item_uuid_ex_1" : 25092144,
                                      "dst_role_1" : 36988224,
                                      "dst_item_uuid_ex_1" : 27381424
                              },
                              "ok" : 1
                      },
                      "mds3" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1085854,
                              "size" : 636901248,
                              "avgObjSize" : 586.5440915629541,
                              "storageSize" : 708362240,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 189837312,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 332133648,
                              "indexSizes" : {
                                      "_id_" : 35238560,
                                      "_sk_1" : 53487392,
                                      "t_when_1" : 30316608,
                                      "op_nuid_1" : 40978112,
                                      "src_role_1" : 62088544,
                                      "src_item_uuid_ex_1" : 25525472,
                                      "dst_role_1" : 56913136,
                                      "dst_item_uuid_ex_1" : 27585824
                              },
                              "ok" : 1
                      }
              },
              "ok" : 1
      }

      2.2 collection stats:

      mongos> db.gm_item.stats()
      {
              "sharded" : true,
              "ns" : "tw2_log_10148.gm_item",
              "count" : 3252213,
              "numExtents" : 48,
              "size" : 1814262528,
              "storageSize" : 1888837632,
              "totalIndexSize" : 1123300640,
              "indexSizes" : {
                      "_id_" : 105543984,
                      "_sk_1" : 158892384,
                      "dst_item_uuid_ex_1" : 103925136,
                      "dst_role_1" : 170216144,
                      "op_nuid_1" : 174484016,
                      "src_item_uuid_ex_1" : 74916688,
                      "src_role_1" : 186363744,
                      "t_when_1" : 148958544
              },
              "avgObjSize" : 557.8547678150231,
              "nindexes" : 8,
              "nchunks" : 60,
              "shards" : {
                      "mds1" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1085111,
                              "size" : 605454048,
                              "avgObjSize" : 557.9650819132789,
                              "storageSize" : 629612544,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 168730624,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 374951360,
                              "indexSizes" : {
                                      "_id_" : 35214032,
                                      "_sk_1" : 53397456,
                                      "t_when_1" : 49775488,
                                      "op_nuid_1" : 58082304,
                                      "src_role_1" : 61974080,
                                      "src_item_uuid_ex_1" : 25116672,
                                      "dst_role_1" : 56815024,
                                      "dst_item_uuid_ex_1" : 34576304
                              },
                              "ok" : 1
                      },
                      "mds3" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1081248,
                              "size" : 602956616,
                              "avgObjSize" : 557.6487688300926,
                              "storageSize" : 629612544,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 168730624,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 373079056,
                              "indexSizes" : {
                                      "_id_" : 35091392,
                                      "_sk_1" : 52023888,
                                      "t_when_1" : 49513856,
                                      "op_nuid_1" : 58433872,
                                      "src_role_1" : 62137600,
                                      "src_item_uuid_ex_1" : 24797808,
                                      "dst_role_1" : 56528864,
                                      "dst_item_uuid_ex_1" : 34551776
                              },
                              "ok" : 1
                      },
                      "mds4" : {
                              "ns" : "tw2_log_10148.gm_item",
                              "count" : 1085854,
                              "size" : 605851864,
                              "avgObjSize" : 557.9496543734241,
                              "storageSize" : 629612544,
                              "numExtents" : 16,
                              "nindexes" : 8,
                              "lastExtentSize" : 168730624,
                              "paddingFactor" : 1,
                              "systemFlags" : 1,
                              "userFlags" : 0,
                              "totalIndexSize" : 375270224,
                              "indexSizes" : {
                                      "_id_" : 35238560,
                                      "_sk_1" : 53471040,
                                      "t_when_1" : 49669200,
                                      "op_nuid_1" : 57967840,
                                      "src_role_1" : 62252064,
                                      "src_item_uuid_ex_1" : 25002208,
                                      "dst_role_1" : 56872256,
                                      "dst_item_uuid_ex_1" : 34797056
                              },
                              "ok" : 1
                      }
              },
              "ok" : 1
      }

      I also tested the hashed shard plan. (the record insert timestamp is used to be a shard key). The insert performance is more terrible. It took 1100 seconds to insert same records.

      Attachments

        Activity

          People

            james.wahlin@mongodb.com James Wahlin
            strlee stronglee
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: