ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Cluster Scalability
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Right now our time estimates work something like this:

      • During cloning: Based on the time already spent cloning, the number of bytes already cloned, and the total number of bytes to clone, extrapolate the time it will take to complete cloning. Report (2 * total time to clone - time already spent cloning) as the current estimate for resharding as a whole. In other words, assume building indexes and applying take exactly as long as cloning did.
      • During building indexes: Report the total time spent cloning as the estimate. This doesn't change as the index builds progress at all.
      • During applying: Compare the wall time of the oplog entries fetched from the donor to the wall time of the recipient to estimate how far behind the donor's oplog tail we are, then report this as the remaining time estimate.

      This has the following problems:

      • The time estimate prior to applying is functionally useless. We assume that cloning and applying take the same amount of time, though this is unlikely to be the case in practice. The time spent cloning is primarily a function of collection size, whereas the time spent applying is primarily a function of write rate.
      • The time estimate today doesn't consider building indexes at all. Our current estimation logic was originally written when building indexes wasn't a distinct phase, but rather done prior to cloning any documents at all. In the previous world, the time spent cloning included the time spent building indexes, but this is no longer the case.
      • The time estimate during applying considers how stale we are, but not the rate at which our staleness is decreasing (which is likely to be faster than realtime; or at least it better be if we ever expect to catch up at all).

      Suggestions for improvement:

      • During cloning, we likely need to use a heuristic based on the collection size, number of indexes, and write rate to estimate how long the index build and applying phases will be.
      • During index builds, we should use the same heuristic for applying, but otherwise estimate based on the actual progress of index builds.
      • During applying, we should consider the rate at which our staleness is decreasing, not just the raw staleness.

            Assignee:
            Unassigned
            Reporter:
            Brett Nawrocki
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: