Investigate ways to improve PBT effectiveness in patches

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Our fast-check property-based tests are seeded, and run a set number of queries. This means that they don't catch all of the bugs they can if they were run for longer. One solution to this is SERVER-91404, where we randomize them in the waterfall so more of the search space is explored.

      We should however make sure we are running the PBTs enough in patches, so that we can prevent as many bugs as possible from being committed (within reason, without having them take too long). It's better to have a test fail before committing than get a BF afterwards.

      This might involve:

      • Tuning the knobs for "numRuns" and "numQueriesPerRun" in the PBTs
      • Sometimes increasing the PBT runtimes, for example if a PBT runs for 5 seconds, maybe it's okay to have it run for 10 seconds
      • We can try to make sure no wasted work is done in the PBTs, so they have more time to run queries
      • We can tune the weights in the models to trigger more bug-prune scenarios. For example if most $match filter out all documents, we can tune it to try to allow more documents which would reach more bug-prune scenarios

      An example improvement that was implemented previously was rewriting the comparator function from JS to C++, since comparing results was a bottleneck. This allowed us to have PBTs run 2x more queries

            Assignee:
            Matt Boros
            Reporter:
            Matt Boros
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: