If would be helpful if we could rely on a piece of information from the mongod logs to synchronize the information in a common timeline.
Sometimes the logs have:
- no timezone information
- clock drift between hosts
which makes it difficult.
One possibility would be to use a unique token or an information from a heartbeat and have the 2 hosts log this info in there corresponding logs. Searching for the string, in both logs, would give you a synchronization point.
- host A records a time and a unique ID
- host A sends the info to host B (may already be available through the heartbeat)
- host A logs the info (the recorded time would be different than the log time, however we care about the first one)
- host B logs the info received, with its own timestamp
- host A or B would add the perceived latency it knows about the other host.
This token should only be needed once per couple of hosts at a frequency of once per hour, or once per day, or once per log file.