[SERVER-25268] OplogBufferCollection should cache last pushed/popped values and tryPop() should not remove documents from collection Created: 25/Jul/16  Updated: 25/Jan/17  Resolved: 20/Sep/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.3.14

Type: Improvement Priority: Major - P3
Reporter: Judah Schvimer Assignee: Benety Goh
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-26033 Add startKeyInclusive option to Inter... Closed
is depended on by SERVER-26191 OplogBufferCollection should batch re... Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 2016-08-29, Repl 2016-09-19, Repl 2016-10-10
Participants:

 Description   

The OplogBufferCollection appears to be about 20x slower than the OplogBufferInMemory.

To improve it, we can:
1) Delete documents in batches in the background
2) Fetch documents and batch them so they are ready when the Applier needs them.

Data can be found here:

https://evergreen.mongodb.com/task/sys_perf_linux_3_node_replSet_initialsync_initialsync_WT_dr_28d04fda95321c84c402338c0849155140dc6cff_16_07_21_19_45_13#/%7B%22compare%22:%5B%7B%22hash%22:%2228d04fda95321c84c402338c0849155140dc6cff%22%7D%5D%7D#%7B%22compare%22:%5B%7B%22hash%22:%2228d04fda95321c84c402338c0849155140dc6cff%22%7D,%7B%22hash%22:%2228d04fda95321c84c402338c0849155140dc6cff%22%7D%5D%7D

Compared to here:

https://evergreen.mongodb.com/task/sys_perf_linux_3_node_replSet_initialsync_initialsync_WT_dr_f4fc210e00ac1be7cc960fa12358e02b0a92ede0_16_07_22_21_36_33#/%7B%22compare%22:%5B%7B%22hash%22:%22f4fc210e00ac1be7cc960fa12358e02b0a92ede0%22%7D%5D%7D



 Comments   
Comment by Githook User [ 20/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 OplogBufferCollection no longer deletes documents from collection on pop
Branch: master
https://github.com/mongodb/mongo/commit/b7cbb1dbc43639349d02dc50f44191b8006b4bcd

Comment by Githook User [ 20/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 added StorageInterfaceImpl::findOne and deleteOne support for startKey and boundInclusion
Branch: master
https://github.com/mongodb/mongo/commit/963d2a0d952a65e84157aa52bc4d71a40fcb9707

Comment by Githook User [ 20/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 create default initial sync oplog buffer with cache for front and back of buffer
Branch: master
https://github.com/mongodb/mongo/commit/0e0e3c8c69ed513a793c6d47a91b1bae932fe50b

Comment by Githook User [ 20/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 added OplogBufferProxy. caches front and back values of the underlying OplogBuffer
Branch: master
https://github.com/mongodb/mongo/commit/79217aff3a1b4be7ec325a802d8601b489efce97

Comment by Githook User [ 16/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 renamed OplogBuffer::blockingPeek to waitForData and removed peek functionality
Branch: master
https://github.com/mongodb/mongo/commit/446c059c3086dcb579816d29b8e1b43da13e8dbb

Comment by Githook User [ 15/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 OplogBufferCollection enforces order of timestamps in documents passed to pushAllNonBlocking with invariant
Branch: master
https://github.com/mongodb/mongo/commit/01448cbf54f1dd5b2345e80eb9251cc903457543

Comment by Githook User [ 15/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 OplogBufferCollection::startup drops existing collection
Branch: master
https://github.com/mongodb/mongo/commit/e7b869930b2d1d520f8e6d41ede2b960fed09510

Comment by Githook User [ 15/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 OplogBufferCollection::pushAllNonBlocking returns early when no documents are provided
Branch: master
https://github.com/mongodb/mongo/commit/a71048d9c095d95185c1a5150d7c8ec545405c19

Comment by Githook User [ 15/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 renamed OplogBufferCollection::_doPop_inlock to _pop_inlock
Branch: master
https://github.com/mongodb/mongo/commit/6815c80e30dc0cf534adc094f1db300f4db6ee43

Comment by Githook User [ 15/Sep/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-25268 changed OplogBuffer::pushAllNonBlocking to return void instead of bool
Branch: master
https://github.com/mongodb/mongo/commit/a0a4b6742ea064a40e51f1441aa2885e6ce40c8a

Comment by Scott Hernandez (Inactive) [ 01/Sep/16 ]

Mathias suggests using the SortedFileWriter which creates temp files in insertion order, has an iterator (more/next) to retrieve documents, and also supports compression. This would allow us to reduce locking and other storage/contention associated with using a collection. It would also mean reduced IO, and allocation in mmapv1 since dropping collections only frees extents not files to the os/fs.

Generated at Thu Feb 08 04:08:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.