Details
-
Task
-
Resolution: Done
-
Major - P3
-
None
-
Fully Compatible
-
Query 14 (05/13/16)
-
0
Description
In build failures, one of these assertions sometimes fails, with values just outside the range deemed acceptable:
// The average random meta value of the first document should be about 0.75. |
ASSERT_GTE(firstTotal / nTrials, 0.74);
|
ASSERT_LTE(firstTotal / nTrials, 0.76);
|
|
|
// The average random meta value of the second document should be about 0.5. |
ASSERT_GTE(secondTotal / nTrials, 0.49);
|
ASSERT_LTE(secondTotal / nTrials, 0.51);
|
After a deep dive into probability theory, Max and I were able to figure out that (assuming a perfect PRNG) the first set of assertions should succeed at least 99.9999% of the time (six 9's). If we double the range, we should get to at least thirteen 9's.
A similar analysis can be applied to the second set of assertions, but it gets messy since it involves taking the convolution of two Beta distributions, which is not fun.