[SERVER-57638] Benchmark Compression of Individual Audit Logs Created: 11/Jun/21  Updated: 29/Jun/21  Resolved: 29/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Shreyas Kalyan Assignee: Adrian Gonzalez Montemayor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File main.py    
Sprint: Security 2021-06-28, Security 2021-07-12
Participants:

 Description   

For PM-2126, we would like to offer customers log compression to reduce the overall size of their logs. However, because of constraints from cloud and concerns regarding data loss, we are unable to compress the entire audit log file. Instead, we want to see if compressing individual audit logs is a viable solution.

In order to determine if it is a viable solution, we would like to see if compression of individual logs provides any considerable benefit for customers. We should write a script that ingests some audit log data, compresses the logs, and compares the sizes of compressed and uncompressed data. Using this tool, we will be able to make a better decision on whether to pursue allowing users to compress individual audit logs.



 Comments   
Comment by Adrian Gonzalez Montemayor [ 29/Jun/21 ]

After investigation we deemed that compression is a worthwhile feature to add to PM-2126.

From investigating user audit log compression we found that compressing JSON audit logs resulted in a ~50% size decrease and BSON audit logs into a ~30% size decrease.

I have attached my script as a source to run these benchmarks in the future.

Generated at Thu Feb 08 05:42:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.