[SERVER-44819] Slow remove on large dataset Created: 25/Nov/19  Updated: 25/Nov/19  Resolved: 25/Nov/19

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 3.6.15
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Alexandru Martin Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Participants:

 Description   

We need to delete a large amount of data (as an example from one of our collections we need to delete approximately 1 billion documents).
 
Every collection has an index(desc) on a createdAt field so we tried:

db.ourCollection.remove({'createdAt': {$lt: ISODate('2019-07-01')}})

Doing this deletes about 1 million documents every hour, which is extremely slow.
 
The replica set is continuously being written to.
 
How can we remove so much data more efficiently and reliably? 
 
v3.6.15
Amazon Linux AMI release 2018.03



 Comments   
Comment by Alexandru Martin [ 25/Nov/19 ]

Do you realize that this is actually a performance bug? So if i re-post with the type changed to BUG, will you politely answer me? 

Comment by Dmitry Agranat [ 25/Nov/19 ]

Hi axiodis,

The SERVER project is for bugs and feature suggestions for the MongoDB server. As this ticket does not appear to be a bug, I will now close it. If you need further assistance with troubleshooting or performance, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Thanks,
Dima

Generated at Thu Feb 08 05:07:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.