Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32082

dropDatabase reports success but db is still on disk of secondary

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.4.6
    • Fix Version/s: None
    • Component/s: Replication, Storage
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      We are using 4 shards to store log data in daily created databases which will be dropped after some days. The reason not to use TTL is that dropDatabase is an instant operation freeing also diskspace immediately. However, the secondaries of 2 of the 4 shards do not wipe the dropped database from disk even though the dropDatabase command reported success.
      The workaround described in SERVER-17397 (dropping the database from within the primary instead mongos) did not delete the database from the secondary event though the command reported success.
      The dropped database is not shown by issuing the command show dbs on the secondary.

      No matter if the command show dbs was issued through mongos, primary or secondary, it shows the same:

      MongoDB server version: 3.4.6
      mongos> show dbs
      admin             0.000GB
      config            0.010GB
      log-2017-11-04   99.273GB
      log-2017-11-05   84.237GB
      log-2017-11-06   96.744GB
      log-2017-11-07   96.260GB
      log-2017-11-08  101.185GB
      log-2017-11-09  109.233GB
      log-2017-11-10  113.048GB
      log-2017-11-11  110.611GB
      log-2017-11-12   99.498GB
      log-2017-11-13  100.777GB
      log-2017-11-14  103.017GB
      log-2017-11-15  118.151GB
      log-2017-11-16  113.948GB
      log-2017-11-17  111.382GB
      log-2017-11-18  124.049GB
      log-2017-11-19  102.645GB
      log-2017-11-20  103.504GB
      log-2017-11-21  120.029GB
      log-2017-11-22  108.096GB
      log-2017-11-23  111.556GB
      log-2017-11-24   50.834GB
      log-2017-11-25    0.000GB
      log-2017-11-26    0.000GB
      log-2017-11-27    0.000GB
      log-2017-11-28    0.000GB
      log-2017-11-29    0.000GB
      log-2017-11-30    0.000GB
      log-2017-12-01    0.000GB
      stats             0.000GB
      test              0.000GB
      mongos>

      These are our shards:

      mongos> sh.status()
      --- Sharding Status ---
        sharding version: {
      	"_id" : 1,
      	"minCompatibleVersion" : 5,
      	"currentVersion" : 6,
      	"clusterId" : ObjectId("598ab847e97e92ba40fa56ee")
      }
        shards:
      	{  "_id" : "offerHistory01",  "host" : "offerHistory01/mongo-016.db00.pro06.eu.idealo.com:27017,mongo-105.db00.pro05.eu.idealo.com:27017",  "state" : 1 }
      	{  "_id" : "offerHistory02",  "host" : "offerHistory02/mongo-119.db00.pro06.eu.idealo.com:27017,mongo-120.db00.pro06.eu.idealo.com:27017",  "state" : 1 }
      	{  "_id" : "offerHistory03",  "host" : "offerHistory03/mongo-121.db00.pro06.eu.idealo.com:27017,mongo-122.db00.pro06.eu.idealo.com:27017",  "state" : 1 }
      	{  "_id" : "offerHistory04",  "host" : "offerHistory04/mongo-047.db00.pro06.eu.idealo.com:27017,mongo-123.db00.pro06.eu.idealo.com:27017",  "state" : 1 }
        active mongoses:
      	"3.4.6" : 5
       autosplit:
      	Currently enabled: yes
        balancer:
      	Currently enabled:  no
      	Currently running:  no


      The balancer is disabled because our shardkey is well balanced (hashed).

      Let's compare the data directories of primary and secondary of the 3rd shard:
      Primary:

      10:58:40]root@mongo-121.db00.pro06 # du -sch /data/offerHistory03/db/*
      176K	/data/offerHistory03/db/admin
      205M	/data/offerHistory03/db/diagnostic.data
      4,2G	/data/offerHistory03/db/journal
      42G	/data/offerHistory03/db/local
      4,0K	/data/offerHistory03/db/log.452017.4507.4522.45zlib
      4,0K	/data/offerHistory03/db/log.452017.4509.4513
      4,0K	/data/offerHistory03/db/log.452017.4509.4527
      4,0K	/data/offerHistory03/db/log.452017.4509.4528
      4,0K	/data/offerHistory03/db/log.452017.4509.4529
      4,0K	/data/offerHistory03/db/log.452017.4509.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4501
      4,0K	/data/offerHistory03/db/log.452017.4510.4502
      4,0K	/data/offerHistory03/db/log.452017.4510.4503
      4,0K	/data/offerHistory03/db/log.452017.4510.4504
      4,0K	/data/offerHistory03/db/log.452017.4510.4505
      4,0K	/data/offerHistory03/db/log.452017.4510.4506
      4,0K	/data/offerHistory03/db/log.452017.4510.4507
      4,0K	/data/offerHistory03/db/log.452017.4510.4508
      4,0K	/data/offerHistory03/db/log.452017.4510.4509
      4,0K	/data/offerHistory03/db/log.452017.4510.4510
      4,0K	/data/offerHistory03/db/log.452017.4510.4511
      4,0K	/data/offerHistory03/db/log.452017.4510.4512
      4,0K	/data/offerHistory03/db/log.452017.4510.4513
      4,0K	/data/offerHistory03/db/log.452017.4510.4514
      4,0K	/data/offerHistory03/db/log.452017.4510.4515
      4,0K	/data/offerHistory03/db/log.452017.4510.4516
      4,0K	/data/offerHistory03/db/log.452017.4510.4517
      4,0K	/data/offerHistory03/db/log.452017.4510.4518
      4,0K	/data/offerHistory03/db/log.452017.4510.4519
      4,0K	/data/offerHistory03/db/log.452017.4510.4520
      4,0K	/data/offerHistory03/db/log.452017.4510.4521
      4,0K	/data/offerHistory03/db/log.452017.4510.4522
      4,0K	/data/offerHistory03/db/log.452017.4510.4523
      4,0K	/data/offerHistory03/db/log.452017.4510.4524
      4,0K	/data/offerHistory03/db/log.452017.4510.4525
      4,0K	/data/offerHistory03/db/log.452017.4510.4526
      4,0K	/data/offerHistory03/db/log.452017.4510.4527
      4,0K	/data/offerHistory03/db/log.452017.4510.4528
      4,0K	/data/offerHistory03/db/log.452017.4510.4529
      4,0K	/data/offerHistory03/db/log.452017.4510.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4531
      4,0K	/data/offerHistory03/db/log.452017.4511.4501
      4,0K	/data/offerHistory03/db/log.452017.4511.4502
      4,0K	/data/offerHistory03/db/log.452017.4511.4503
      25G	/data/offerHistory03/db/log.452017.4511.4504
      22G	/data/offerHistory03/db/log.452017.4511.4505
      25G	/data/offerHistory03/db/log.452017.4511.4506
      25G	/data/offerHistory03/db/log.452017.4511.4507
      26G	/data/offerHistory03/db/log.452017.4511.4508
      28G	/data/offerHistory03/db/log.452017.4511.4509
      29G	/data/offerHistory03/db/log.452017.4511.4510
      28G	/data/offerHistory03/db/log.452017.4511.4511
      25G	/data/offerHistory03/db/log.452017.4511.4512
      26G	/data/offerHistory03/db/log.452017.4511.4513
      26G	/data/offerHistory03/db/log.452017.4511.4514
      30G	/data/offerHistory03/db/log.452017.4511.4515
      29G	/data/offerHistory03/db/log.452017.4511.4516
      28G	/data/offerHistory03/db/log.452017.4511.4517
      32G	/data/offerHistory03/db/log.452017.4511.4518
      26G	/data/offerHistory03/db/log.452017.4511.4519
      26G	/data/offerHistory03/db/log.452017.4511.4520
      30G	/data/offerHistory03/db/log.452017.4511.4521
      28G	/data/offerHistory03/db/log.452017.4511.4522
      28G	/data/offerHistory03/db/log.452017.4511.4523
      13G	/data/offerHistory03/db/log.452017.4511.4524
      20K	/data/offerHistory03/db/log.452017.4511.4525
      20K	/data/offerHistory03/db/log.452017.4511.4526
      20K	/data/offerHistory03/db/log.452017.4511.4527
      20K	/data/offerHistory03/db/log.452017.4511.4528
      20K	/data/offerHistory03/db/log.452017.4511.4529
      20K	/data/offerHistory03/db/log.452017.4511.4530
      20K	/data/offerHistory03/db/log.452017.4512.4501
      36K	/data/offerHistory03/db/_mdb_catalog.wt
      4,0K	/data/offerHistory03/db/mongod.lock
      4,0K	/data/offerHistory03/db/perf
      4,0K	/data/offerHistory03/db/perf2
      64K	/data/offerHistory03/db/sizeStorer.wt
      4,0K	/data/offerHistory03/db/storage.bson
      4,0K	/data/offerHistory03/db/test
      4,0K	/data/offerHistory03/db/_tmp
      4,0K	/data/offerHistory03/db/WiredTiger
      4,0K	/data/offerHistory03/db/WiredTigerLAS.wt
      4,0K	/data/offerHistory03/db/WiredTiger.lock
      4,0K	/data/offerHistory03/db/WiredTiger.turtle
      452K	/data/offerHistory03/db/WiredTiger.wt
      591G	insgesamt
      


      Secondary:

      [10:58:40]root@mongo-122.db00.pro06 # du -sch /data/offerHistory03/db/*
      176K	/data/offerHistory03/db/admin
      194M	/data/offerHistory03/db/diagnostic.data
      400M	/data/offerHistory03/db/journal
      42G	/data/offerHistory03/db/local
      4,0K	/data/offerHistory03/db/log.452017.4507.4522.45zlib
      4,0K	/data/offerHistory03/db/log.452017.4509.4513
      4,0K	/data/offerHistory03/db/log.452017.4509.4527
      4,0K	/data/offerHistory03/db/log.452017.4509.4528
      4,0K	/data/offerHistory03/db/log.452017.4509.4529
      4,0K	/data/offerHistory03/db/log.452017.4509.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4501
      4,0K	/data/offerHistory03/db/log.452017.4510.4502
      4,0K	/data/offerHistory03/db/log.452017.4510.4503
      4,0K	/data/offerHistory03/db/log.452017.4510.4504
      4,0K	/data/offerHistory03/db/log.452017.4510.4505
      4,0K	/data/offerHistory03/db/log.452017.4510.4506
      4,0K	/data/offerHistory03/db/log.452017.4510.4507
      4,0K	/data/offerHistory03/db/log.452017.4510.4508
      4,0K	/data/offerHistory03/db/log.452017.4510.4509
      4,0K	/data/offerHistory03/db/log.452017.4510.4510
      4,0K	/data/offerHistory03/db/log.452017.4510.4511
      4,0K	/data/offerHistory03/db/log.452017.4510.4512
      4,0K	/data/offerHistory03/db/log.452017.4510.4513
      4,0K	/data/offerHistory03/db/log.452017.4510.4514
      4,0K	/data/offerHistory03/db/log.452017.4510.4515
      4,0K	/data/offerHistory03/db/log.452017.4510.4516
      4,0K	/data/offerHistory03/db/log.452017.4510.4517
      4,0K	/data/offerHistory03/db/log.452017.4510.4518
      4,0K	/data/offerHistory03/db/log.452017.4510.4519
      4,0K	/data/offerHistory03/db/log.452017.4510.4520
      4,0K	/data/offerHistory03/db/log.452017.4510.4521
      4,0K	/data/offerHistory03/db/log.452017.4510.4522
      4,0K	/data/offerHistory03/db/log.452017.4510.4523
      4,0K	/data/offerHistory03/db/log.452017.4510.4524
      4,0K	/data/offerHistory03/db/log.452017.4510.4525
      4,0K	/data/offerHistory03/db/log.452017.4510.4526
      4,0K	/data/offerHistory03/db/log.452017.4510.4527
      4,0K	/data/offerHistory03/db/log.452017.4510.4528
      4,0K	/data/offerHistory03/db/log.452017.4510.4529
      4,0K	/data/offerHistory03/db/log.452017.4510.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4531
      4,0K	/data/offerHistory03/db/log.452017.4511.4501
      17G	/data/offerHistory03/db/log.452017.4511.4502
      22G	/data/offerHistory03/db/log.452017.4511.4503
      25G	/data/offerHistory03/db/log.452017.4511.4504
      22G	/data/offerHistory03/db/log.452017.4511.4505
      25G	/data/offerHistory03/db/log.452017.4511.4506
      25G	/data/offerHistory03/db/log.452017.4511.4507
      26G	/data/offerHistory03/db/log.452017.4511.4508
      28G	/data/offerHistory03/db/log.452017.4511.4509
      28G	/data/offerHistory03/db/log.452017.4511.4510
      28G	/data/offerHistory03/db/log.452017.4511.4511
      25G	/data/offerHistory03/db/log.452017.4511.4512
      26G	/data/offerHistory03/db/log.452017.4511.4513
      26G	/data/offerHistory03/db/log.452017.4511.4514
      30G	/data/offerHistory03/db/log.452017.4511.4515
      29G	/data/offerHistory03/db/log.452017.4511.4516
      28G	/data/offerHistory03/db/log.452017.4511.4517
      32G	/data/offerHistory03/db/log.452017.4511.4518
      26G	/data/offerHistory03/db/log.452017.4511.4519
      26G	/data/offerHistory03/db/log.452017.4511.4520
      31G	/data/offerHistory03/db/log.452017.4511.4521
      27G	/data/offerHistory03/db/log.452017.4511.4522
      28G	/data/offerHistory03/db/log.452017.4511.4523
      13G	/data/offerHistory03/db/log.452017.4511.4524
      20K	/data/offerHistory03/db/log.452017.4511.4525
      20K	/data/offerHistory03/db/log.452017.4511.4526
      20K	/data/offerHistory03/db/log.452017.4511.4527
      20K	/data/offerHistory03/db/log.452017.4511.4528
      20K	/data/offerHistory03/db/log.452017.4511.4529
      20K	/data/offerHistory03/db/log.452017.4511.4530
      20K	/data/offerHistory03/db/log.452017.4512.4501
      40K	/data/offerHistory03/db/_mdb_catalog.wt
      4,0K	/data/offerHistory03/db/mongod.lock
      4,0K	/data/offerHistory03/db/perf
      4,0K	/data/offerHistory03/db/perf2
      64K	/data/offerHistory03/db/sizeStorer.wt
      4,0K	/data/offerHistory03/db/storage.bson
      4,0K	/data/offerHistory03/db/test
      4,0K	/data/offerHistory03/db/WiredTiger
      4,0K	/data/offerHistory03/db/WiredTigerLAS.wt
      4,0K	/data/offerHistory03/db/WiredTiger.lock
      4,0K	/data/offerHistory03/db/WiredTiger.turtle
      384K	/data/offerHistory03/db/WiredTiger.wt
      625G	insgesamt
      


      You can see that the secondary still has data for databases log-2017-11-02 and log-2017-11-03:

      17G	/data/offerHistory03/db/log.452017.4511.4502
      22G	/data/offerHistory03/db/log.452017.4511.4503


      whereas the primary deleted these database correctly from disk:

      4,0K	/data/offerHistory03/db/log.452017.4511.4502
      4,0K	/data/offerHistory03/db/log.452017.4511.4503

      Our workaround is to restart the mongod process of the secondary. Once restarted, the orphaned databases are successfully deleted from disk:

      [11:05:57]root@mongo-122.db00.pro06 # du -sch /data/offerHistory03/db/*
      176K	/data/offerHistory03/db/admin
      194M	/data/offerHistory03/db/diagnostic.data
      271M	/data/offerHistory03/db/journal
      42G	/data/offerHistory03/db/local
      4,0K	/data/offerHistory03/db/log.452017.4507.4522.45zlib
      4,0K	/data/offerHistory03/db/log.452017.4509.4513
      4,0K	/data/offerHistory03/db/log.452017.4509.4527
      4,0K	/data/offerHistory03/db/log.452017.4509.4528
      4,0K	/data/offerHistory03/db/log.452017.4509.4529
      4,0K	/data/offerHistory03/db/log.452017.4509.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4501
      4,0K	/data/offerHistory03/db/log.452017.4510.4502
      4,0K	/data/offerHistory03/db/log.452017.4510.4503
      4,0K	/data/offerHistory03/db/log.452017.4510.4504
      4,0K	/data/offerHistory03/db/log.452017.4510.4505
      4,0K	/data/offerHistory03/db/log.452017.4510.4506
      4,0K	/data/offerHistory03/db/log.452017.4510.4507
      4,0K	/data/offerHistory03/db/log.452017.4510.4508
      4,0K	/data/offerHistory03/db/log.452017.4510.4509
      4,0K	/data/offerHistory03/db/log.452017.4510.4510
      4,0K	/data/offerHistory03/db/log.452017.4510.4511
      4,0K	/data/offerHistory03/db/log.452017.4510.4512
      4,0K	/data/offerHistory03/db/log.452017.4510.4513
      4,0K	/data/offerHistory03/db/log.452017.4510.4514
      4,0K	/data/offerHistory03/db/log.452017.4510.4515
      4,0K	/data/offerHistory03/db/log.452017.4510.4516
      4,0K	/data/offerHistory03/db/log.452017.4510.4517
      4,0K	/data/offerHistory03/db/log.452017.4510.4518
      4,0K	/data/offerHistory03/db/log.452017.4510.4519
      4,0K	/data/offerHistory03/db/log.452017.4510.4520
      4,0K	/data/offerHistory03/db/log.452017.4510.4521
      4,0K	/data/offerHistory03/db/log.452017.4510.4522
      4,0K	/data/offerHistory03/db/log.452017.4510.4523
      4,0K	/data/offerHistory03/db/log.452017.4510.4524
      4,0K	/data/offerHistory03/db/log.452017.4510.4525
      4,0K	/data/offerHistory03/db/log.452017.4510.4526
      4,0K	/data/offerHistory03/db/log.452017.4510.4527
      4,0K	/data/offerHistory03/db/log.452017.4510.4528
      4,0K	/data/offerHistory03/db/log.452017.4510.4529
      4,0K	/data/offerHistory03/db/log.452017.4510.4530
      4,0K	/data/offerHistory03/db/log.452017.4510.4531
      4,0K	/data/offerHistory03/db/log.452017.4511.4501
      4,0K	/data/offerHistory03/db/log.452017.4511.4502
      4,0K	/data/offerHistory03/db/log.452017.4511.4503
      25G	/data/offerHistory03/db/log.452017.4511.4504
      22G	/data/offerHistory03/db/log.452017.4511.4505
      25G	/data/offerHistory03/db/log.452017.4511.4506
      25G	/data/offerHistory03/db/log.452017.4511.4507
      26G	/data/offerHistory03/db/log.452017.4511.4508
      28G	/data/offerHistory03/db/log.452017.4511.4509
      28G	/data/offerHistory03/db/log.452017.4511.4510
      28G	/data/offerHistory03/db/log.452017.4511.4511
      25G	/data/offerHistory03/db/log.452017.4511.4512
      26G	/data/offerHistory03/db/log.452017.4511.4513
      26G	/data/offerHistory03/db/log.452017.4511.4514
      30G	/data/offerHistory03/db/log.452017.4511.4515
      29G	/data/offerHistory03/db/log.452017.4511.4516
      28G	/data/offerHistory03/db/log.452017.4511.4517
      32G	/data/offerHistory03/db/log.452017.4511.4518
      26G	/data/offerHistory03/db/log.452017.4511.4519
      26G	/data/offerHistory03/db/log.452017.4511.4520
      31G	/data/offerHistory03/db/log.452017.4511.4521
      27G	/data/offerHistory03/db/log.452017.4511.4522
      28G	/data/offerHistory03/db/log.452017.4511.4523
      13G	/data/offerHistory03/db/log.452017.4511.4524
      20K	/data/offerHistory03/db/log.452017.4511.4525
      20K	/data/offerHistory03/db/log.452017.4511.4526
      20K	/data/offerHistory03/db/log.452017.4511.4527
      20K	/data/offerHistory03/db/log.452017.4511.4528
      20K	/data/offerHistory03/db/log.452017.4511.4529
      20K	/data/offerHistory03/db/log.452017.4511.4530
      20K	/data/offerHistory03/db/log.452017.4512.4501
      40K	/data/offerHistory03/db/_mdb_catalog.wt
      4,0K	/data/offerHistory03/db/mongod.lock
      4,0K	/data/offerHistory03/db/perf
      4,0K	/data/offerHistory03/db/perf2
      64K	/data/offerHistory03/db/sizeStorer.wt
      4,0K	/data/offerHistory03/db/storage.bson
      4,0K	/data/offerHistory03/db/test
      4,0K	/data/offerHistory03/db/WiredTiger
      4,0K	/data/offerHistory03/db/WiredTigerLAS.wt
      4,0K	/data/offerHistory03/db/WiredTiger.lock
      4,0K	/data/offerHistory03/db/WiredTiger.turtle
      384K	/data/offerHistory03/db/WiredTiger.wt
      588G	insgesamt

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: