[SERVER-30252] Write oplog operations to kafka Created: 21/Jul/17 Updated: 21/Jul/17 Resolved: 21/Jul/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Jose Luis Pedrosa | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Hi When insert/update/delete operation is performed, write it not only to the oplog, but also to a kafka endpoint. The shard name and the collection could be used to determine the kafka partition number. Benefits/Use cases:
Despite this could be implemented via external scripts, they become complicated due to the fact that would need to also interact with the config servers (or mongos commands, not sure if all the info is available). To detect new nodes in the cluster, also scaling the script/monitor software can become a difficult task while implemented on the mongod process seems simple. Potentially it even could be used to replicate the DBs instead of the oplog, and in the event of the failure of the primary, secondaries would be configured to process the pending oplog messages in kafka before actually marking the mongod instance as online.As kafka is replicated and HA,if the operation made it in the primary we can be sure it can make it to the secondaries. At the end, kafka is a write ahead log, same as oplog. JL |
| Comments |
| Comment by Ramon Fernandez Marina [ 21/Jul/17 ] |
|
There are previous tickets that would enable this: Arguably the MongoDB server should not be in charge of feeding other systems, but just enabling other tools to interface with it in a way that permits doing something like what you describe. I'm therefore going to close this request as a duplicate. I'd encourage you to watch and vote for the tickets above, so your particular use case can be taken into consideration during the design phase. Regards, |