Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44819

Slow remove on large dataset

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 3.6.15
    • Performance
    • None
    • Fully Compatible

    Description

      We need to delete a large amount of data (as an example from one of our collections we need to delete approximately 1 billion documents).
       
      Every collection has an index(desc) on a createdAt field so we tried:

      db.ourCollection.remove({'createdAt': {$lt: ISODate('2019-07-01')}})

      Doing this deletes about 1 million documents every hour, which is extremely slow.
       
      The replica set is continuously being written to.
       
      How can we remove so much data more efficiently and reliably? 
       
      v3.6.15
      Amazon Linux AMI release 2018.03

      Attachments

        Activity

          People

            Unassigned Unassigned
            axiodis Alexandru Martin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: