Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-53870

Improve view creation performance over time

XMLWordPrintableJSON

    • Fully Compatible
    • v6.0, v5.0, v4.4
    • Execution Team 2021-12-13, Execution Team 2021-12-27, Execution Team 2022-01-10, Execution Team 2022-01-24, Execution Team 2022-02-07, Execution Team 2022-02-21, Execution Team 2022-03-07, Execution Team 2022-03-21, Execution Team 2022-04-04, Execution Team 2022-04-18, Execution Team 2022-05-02
    • 105

      Even though view creation is rather fast, in my experiment it took 100 seconds to create 5000 views, view creation degrades over time. Currently, we take collection MODE_X lock and need to reload the view catalog during each view creation.

      The script I used is a single-threaded view creation but running the same script against different collections in parallel does not improve the total execution time as it seems that reloading the view catalog is the bottleneck.

      Repro:

      Unable to find source-code formatter for language: javascript. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      function main() {
    var views_to_create = 5000;
    print("Starting...");
    db = db.getSiblingDB("views_db");
    db.dropDatabase();
    print("DB Dropped");
    var coll_name = "my_collection";
    db.createCollection(coll_name);
     print("DB/Collection Created");
     var results=[];
     for (var i=0; i < views_to_create; i=i+1){
         var field = 'field_' + Math.floor(Math.random() * 10000).toString();
         var pipeline = [{'$project': { }}];
         pipeline[0]['$project'][field] = '$a';
         var start = new Date();
         db.runCommand({"create": "view_" + i,
            "viewOn": coll_name, 
            "pipeline": pipeline});
         var end =  new Date();
        if (i % 100 == 0){
            print ((end-start) + " taken for creating view " + i);
        }
     }
}
main();

        1. diagnostic.data_Jan21_perf_gdb.tar.gz
          6.62 MB
        2. gdb_views.tar.gz
          41 kB
        3. mongodb_Jan21_perf_gdb.log.tar.gz
          251 kB
        4. perf_views.tar.gz
          1.11 MB
        5. Screenshot 2021-01-19 at 15.54.01.png
          Screenshot 2021-01-19 at 15.54.01.png
          30 kB

            Assignee:
            shinyee.tan@mongodb.com Shin Yee Tan
            Reporter:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: