Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20961

Large amounts of create and drop collections can cause listDatabases to be slow under WiredTiger

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.0.6, 3.0.7, 3.2.0-rc0
    • Fix Version/s: 3.0.9, 3.2.0-rc4
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      require "mongo"
       
      client1 = Mongo::Client.new('mongodb://127.0.0.1:27017/')
      client2 = Mongo::Client.new('mongodb://127.0.0.1:27017/')
      client3 = Mongo::Client.new('mongodb://127.0.0.1:27017/')
      client4 = Mongo::Client.new('mongodb://127.0.0.1:27017/')
       
      Mongo::Logger.logger.level = Logger::WARN
       
      for x in 0..100
          thr = Thread.new {
      #        sleep 1
              before = Time.now
              client1.use("admin").command(:listDatabases => 1)
              after= Time.now
              p "took: " + (after-before).to_s
          }
              thr1 = Thread.new {
              for i in 0..100
                      name = "abcd" + ((x*100)+i).to_s;
                      client2.use(name)[:foo].indexes.create_one({ :x => 1})
                      client2.use(name)[:foo].insert_one({ :x => 1})
                      client2.use(name)[:baz].indexes.create_one({ :x => 1})
                      client2.use(name)[:bar].insert_one({ :x => 1})
              end
              }
              thr2 = Thread.new {
              for i in 0..100
                      name = "abcd" + ((x*100)+i).to_s;
                      client3.use(name)[:bar].indexes.create_one({ :x => 1})
                      client3.use(name)[:bar].insert_one({ :x => 1})
                      client3.use(name)[:qux].indexes.create_one({ :x => 1})
                      client3.use(name)[:qux].insert_one({ :x => 1})
              end
              }
              thr3 = Thread.new {
              for i in 0..100
                      name = "abcd" + ((x*100)+i).to_s;
                      client4.use(name)[:alpha].indexes.create_one({ :x => 1})
                      client4.use(name)[:alpha].insert_one({ :x => 1})
                      client4.use(name)[:beta].indexes.create_one({ :x => 1})
                      client4.use(name)[:beta].insert_one({ :x => 1})
              end
              }
              thr1.join
              thr2.join
              thr3.join
              thr4 = Thread.new {
              for i in 0..100
                      name = "abcd" + ((x*100)+i).to_s;
                      client2.use(name)[:foo].drop
                      client3.use(name)[:baz].drop
                      client4.use(name)[:beta].drop
              end
              }
              thr4.join
          sleep 20
          thr.join()
      end
      

      Show
      require "mongo"   client1 = Mongo::Client. new ( 'mongodb://127.0.0.1:27017/' ) client2 = Mongo::Client. new ( 'mongodb://127.0.0.1:27017/' ) client3 = Mongo::Client. new ( 'mongodb://127.0.0.1:27017/' ) client4 = Mongo::Client. new ( 'mongodb://127.0.0.1:27017/' )   Mongo::Logger.logger.level = Logger:: WARN   for x in 0 .. 100 thr = Thread . new { # sleep 1 before = Time .now client1.use( "admin" ).command( :listDatabases => 1 ) after= Time .now p "took: " + (after-before).to_s } thr1 = Thread . new { for i in 0 .. 100 name = "abcd" + ((x* 100 )+i).to_s; client2.use(name)[ :foo ].indexes.create_one({ :x => 1 }) client2.use(name)[ :foo ].insert_one({ :x => 1 }) client2.use(name)[ :baz ].indexes.create_one({ :x => 1 }) client2.use(name)[ :bar ].insert_one({ :x => 1 }) end } thr2 = Thread . new { for i in 0 .. 100 name = "abcd" + ((x* 100 )+i).to_s; client3.use(name)[ :bar ].indexes.create_one({ :x => 1 }) client3.use(name)[ :bar ].insert_one({ :x => 1 }) client3.use(name)[ :qux ].indexes.create_one({ :x => 1 }) client3.use(name)[ :qux ].insert_one({ :x => 1 }) end } thr3 = Thread . new { for i in 0 .. 100 name = "abcd" + ((x* 100 )+i).to_s; client4.use(name)[ :alpha ].indexes.create_one({ :x => 1 }) client4.use(name)[ :alpha ].insert_one({ :x => 1 }) client4.use(name)[ :beta ].indexes.create_one({ :x => 1 }) client4.use(name)[ :beta ].insert_one({ :x => 1 }) end } thr1.join thr2.join thr3.join thr4 = Thread . new { for i in 0 .. 100 name = "abcd" + ((x* 100 )+i).to_s; client2.use(name)[ :foo ].drop client3.use(name)[ :baz ].drop client4.use(name)[ :beta ].drop end } thr4.join sleep 20 thr.join() end

      Description

      Under a workload with a high volume of collection (and index) creation over a number of database then the dropping of these collections will cause listDatabases command to run longer on a WT instance.

      Using the reproduction script shows the following execution time for listDatabases per lap in seconds:

      "took: 1.705329872"
      "took: 1.328905619"
      "took: 1.55636039"
      "took: 1.221779232"
      "took: 1.772605573"
      "took: 1.689325783"
      "took: 1.15700596"
      "took: 1.470458173"
      "took: 1.255596249"
      "took: 1.455441504"
      "took: 8.487761074"
      "took: 13.784377947"
      "took: 15.079964722"
      "took: 13.892499541"
      "took: 15.179368285"
      "took: 14.683478172"
      "took: 14.776398389"
      "took: 14.481528086"
      "took: 14.688469802"
      "took: 13.738872847"
      "took: 14.298077263"
      

        Attachments

        1. threads
          33 kB
        2. threads2
          33 kB

          Issue Links

            Activity

              People

              • Votes:
                2 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: