Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21460

WiredTiger primaries hold on to deleted collection data and index files

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.6
    • Component/s: WiredTiger
    • Labels:
      None

      We are running v3.0.6 mongoD binaries. In one of our clusters, we dropped a large collection last week. I noticed today that all the primaries in the cluster had significantly higher disk utilization than the secondaries, so I tried stepping down a primary. Upon stepping down, the disk util immediately dropped to the level of the secondaries. The output of du also shows that after stepping down, a collection file along with several index files had been deleted. Note the mongoD process was not restarted at all.

      Before:

      du -sm * | sort -n
      1	collection-0-8259577013043377846.wt
      1	collection-11-8259577013043377846.wt
      1	collection-13-8259577013043377846.wt
      1	collection-17-8259577013043377846.wt
      1	collection-19-8259577013043377846.wt
      1	collection-2-8259577013043377846.wt
      1	collection-4-8259577013043377846.wt
      1	collection-7-8259577013043377846.wt
      1	index-12-8259577013043377846.wt
      1	index-14-8259577013043377846.wt
      1	index-1-8259577013043377846.wt
      1	index-18-8259577013043377846.wt
      1	index-20-8259577013043377846.wt
      1	index-29-8259577013043377846.wt
      1	index-3-8259577013043377846.wt
      1	index-5-8259577013043377846.wt
      1	index-8-8259577013043377846.wt
      1	_mdb_catalog.wt
      1	mongod.lock
      1	sizeStorer.wt
      1	storage.bson
      1	_tmp
      1	WiredTiger
      1	WiredTiger.basecfg
      1	WiredTiger.lock
      1	WiredTiger.turtle
      1	WiredTiger.wt
      6	index-35-8259577013043377846.wt
      7	index-37-8259577013043377846.wt
      8	index-36-8259577013043377846.wt
      10	index-26-8259577013043377846.wt
      38	collection-25-8259577013043377846.wt
      67	index-24-8259577013043377846.wt
      76	collection-23-8259577013043377846.wt
      77	index-16-8259577013043377846.wt
      78	index-31-8259577013043377846.wt
      94	index-33-8259577013043377846.wt
      166	index-32-8259577013043377846.wt
      187	index-30-8259577013043377846.wt
      201	journal
      205	index-34-8259577013043377846.wt
      246	moveChunk
      256	index-22-8259577013043377846.wt
      616	collection-21-8259577013043377846.wt
      2814	collection-6-8259577013043377846.wt
      3732	index-10-8259577013043377846.wt
      4137	index-28-8259577013043377846.wt
      5641	index-27-8259577013043377846.wt
      10368	collection-9-8259577013043377846.wt
      12011	collection-15-8259577013043377846.wt
      

      After:

      sudo du -sm * | sort -n
      1	collection-0-8259577013043377846.wt
      1	collection-11-8259577013043377846.wt
      1	collection-13-8259577013043377846.wt
      1	collection-17-8259577013043377846.wt
      1	collection-19-8259577013043377846.wt
      1	collection-2-8259577013043377846.wt
      1	collection-4-8259577013043377846.wt
      1	collection-7-8259577013043377846.wt
      1	index-12-8259577013043377846.wt
      1	index-14-8259577013043377846.wt
      1	index-1-8259577013043377846.wt
      1	index-18-8259577013043377846.wt
      1	index-20-8259577013043377846.wt
      1	index-29-8259577013043377846.wt
      1	index-3-8259577013043377846.wt
      1	index-5-8259577013043377846.wt
      1	index-8-8259577013043377846.wt
      1	_mdb_catalog.wt
      1	mongod.lock
      1	sizeStorer.wt
      1	storage.bson
      1	_tmp
      1	WiredTiger
      1	WiredTiger.basecfg
      1	WiredTiger.lock
      1	WiredTiger.turtle
      1	WiredTiger.wt
      6	index-35-8259577013043377846.wt
      7	index-37-8259577013043377846.wt
      8	index-36-8259577013043377846.wt
      10	index-26-8259577013043377846.wt
      38	collection-25-8259577013043377846.wt
      67	index-24-8259577013043377846.wt
      76	collection-23-8259577013043377846.wt
      77	index-16-8259577013043377846.wt
      78	index-31-8259577013043377846.wt
      94	index-33-8259577013043377846.wt
      166	index-32-8259577013043377846.wt
      187	index-30-8259577013043377846.wt
      201	journal
      205	index-34-8259577013043377846.wt
      246	moveChunk
      256	index-22-8259577013043377846.wt
      616	collection-21-8259577013043377846.wt
      2814	collection-6-8259577013043377846.wt
      12011	collection-15-8259577013043377846.wt
      

      Is this expected behavior? Do we need to step down all primaries after dropping collections or indexes to reclaim disk space?

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            dai@foursquare.com Dai Shi
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: