[SERVER-16947] Allow db server to write oplog data to another server and collection instead of writing to local db Created: 20/Jan/15  Updated: 15/Feb/15  Resolved: 15/Feb/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Minor - P4
Reporter: Gianfranco Palumbo Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Participants:

 Description   

In heavy write dbs, we see performance drop when oplog is enabled (Master-Slave, MS or RS).

In case mongod would allow options f.e.

--sendoplogserver [server:port] --sendoplogdb [db] --sendoplogcollection [collection]

this would help the master database to focus just on application needs (to write data) and less overhead would be needed for local writes of the oplog.

Assuming 4 heavy-write servers. It would be cheap to build one slow server on fast network to receive ALL oplogs from the 4 servers.



 Comments   
Comment by Eliot Horowitz (Inactive) [ 15/Feb/15 ]

A key requirement for the oplog is that the data and oplog entry are done transactionally.
Don't think we're going to want to be doing 2 phase commits for the oplog.
Lots of other options for improving oplog performance though, which is always going to be something we work on.

Comment by marian badinka [ 28/Jan/15 ]

Hi Eric,

I do not think the "serializing" of writes via socket is an issue, because it happens on daily basis when the Slaves are reading the oplog from Master via network - what is serial transmission.

I also think network latency is also not an issue and if a user/admin/architect could choose if to use local oplog or remote oplog, this would increase the "creativity" and allow to use mongodb in many more use cases.

Regards \ marian

Comment by Eric Milkie [ 27/Jan/15 ]

An interesting idea. In your example, you would be making the oplog server a single point of failure for all 4 servers if you did that, though. Also, it would be super easy to corrupt things if you provided an incorrect parameter to your proposed command line flags. Finally, I'm not sure it would help much, since you would either need to stream the writes through one socket (thus serializing writes the same as when writing locally, except possibly with worse performance), or you would need to use multiple sockets, one per incoming user connection, which may not scale well.

Generated at Thu Feb 08 03:42:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.