[SERVER-41801] [POC] Make Initial sync collection cloner to use wiredTiger Bulk-load api Created: 17/Jun/19  Updated: 29/Oct/23  Resolved: 15/Aug/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.3.1

Type: Task Priority: Major - P3
Reporter: Suganthi Mani Assignee: Vishnu Kaushik
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-16206 Use the WT bulk loader for collection... Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 2019-07-15, Repl 2019-07-29, Repl 2019-08-12, Repl 2019-08-26
Participants:

 Description   

Use wiredTiger Bulk-load api to insert documents into the collection table during initial sync collection phase.



 Comments   
Comment by Alexander Gorrod [ 09/Aug/19 ]

vishnu.kaushik you asked me a question in slack about using Python to test the performance of the bulk load API:

I’m working on the Replication Team in NYC and trying to measure the speedup from using a WT bulk cursor for initial sync. While we have some stats illustrating the speedup from using a bulk cursor in the context of a mongod initial sync, we’d like to see the speed gain of a bulk cursor on WT in isolation.

We built WT and used this file (https://github.com/wiredtiger/wiredtiger/blob/master/test/suite/test_bulk01.py) to measure the speedup difference. Long story short, when ran the Python tests, we saw a 1.3x improvement (rather than a 2x to 3x improvement), but we suspect that this may be due to Python itself being slow.

I agree - the primary overhead in that test is in Python - not in the WiredTiger code.

Do you have any tests that illustrate the speed difference between a bulk cursor and a normal cursor?

We have an internal WiredTiger benchmarking tool that suits your needs better. It's called wtperf. Compiling standalone WiredTiger builds a wtperf binary by default, so it should be in your build tree. The load phase of wtperf uses bulk load if it's single threaded, so the following configuration file will give you results of how long a bulk load of 10 million records takes:

conn_config="cache_size=500MB"                                                  
table_config="type=file"                                                        
icount=10000000                                                                 
report_interval=5                                                               
run_time=0                                                                      
populate_threads=1

The simplest way to see a result you can compare without bulk is to start using a multithreaded load phase:

conn_config="cache_size=500MB"                                                  
table_config="type=file"                                                        
icount=10000000                                                                 
report_interval=5                                                               
run_time=0                                                                      
populate_threads=2

My quick test with those configurations shows that the bulk load takes ~9 seconds and the non-bulk took 29. Let me know if you have questions about my suggestion or about how to run wtperf.

Generated at Thu Feb 08 04:58:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.