[SERVER-54713] Oplog compression ratio Created: 22/Feb/21  Updated: 22/Feb/21  Resolved: 22/Feb/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Bowen Liu Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

We're running a mongodb 4.2  with zstd collection block compression. The oplog is set to oplogSizeMB: 3000000 MB, but the actual collection size is ~350GB. It's like 10X compression ration. According to https://source.wiredtiger.com/3.1.0/compression.html, the default compression ration for zstd is 3. What explains the big difference between the data size and total size of oplog collection?

 

PRIMARY> db.getReplicationInfo()
{
	"logSizeMB" : 3000000,
	"usedMB" : 2996131.66,
	"timeDiff" : 123687,
	"timeDiffHours" : 34.36,
	"tFirst" : "Sun Feb 21 2021 01:55:32 GMT-0700 (MST)",
	"tLast" : "Mon Feb 22 2021 12:16:59 GMT-0700 (MST)",
	"now" : "Mon Feb 22 2021 12:16:59 GMT-0700 (MST)"
}
 
PRIMARY> db.oplog.rs.totalSize()
379049607168
 
PRIMARY> db.oplog.rs.dataSize()
NumberLong("3118230005927") 

 



 Comments   
Comment by Eric Sedor [ 22/Feb/21 ]

Hi bliu@llnw.com,

It's worth noting that totalSize() is derived from the storage size on disk (where compression will be a factor) but dataSize() does not and so will not take compression into account. These are shell helper methods that draw from the collStats command.

That said, the SERVER project is for bugs and feature suggestions for the MongoDB server. For further discussion around interpreting these fields or understanding what compression ratio you're getting, we'd like to encourage you to start by asking our community for help by posting on the MongoDB Developer Community Forums.

If the discussion there leads you to suspect a bug in the MongoDB server, then we'd want to discuss here in the SERVER project.

Thanks,
Eric

Comment by Bowen Liu [ 22/Feb/21 ]

Please correct the title to oplog compression ratio.

Generated at Thu Feb 08 05:34:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.