Our fuzzers are designed to alert us of any change in behavior between different versions or configurations of mongo. They currently generate a BFG every time a query fails on one version and succeeds on another. Often, these are invalid queries that contain some improper value or operation. However, the offending part of the query may be optimized away on one side before it has a chance to error; or it may trigger a different error first because it executes in a different order.

The problem increased sharply when we turned the SBE engine on by default. Since SBE re-implements MQL expressions from scratch, there are many differences in undocumented behavior from non-SBE versions of mongo. So far, we have addressed these BFs by adding workarounds to the fuzzer to suppress BFGs in specific scenarios. The list of these workarounds in the fuzzers is growing long.

We dedicate a lot of time and resources to these BFs. Many of them are connected to queries that customers should not be running, since they are invalid MQL. Our documentation does not promise consistent behavior for invalid queries.

• How likely are customers to ever suffer as a result of these behavior changes in practice?
• How much value does it provide to spend developer time resolving these types of differences?
• What behavior changes do we want to prioritize identifying and fixing?
• How can we most efficiently change the fuzzers to reflect these priorities?

One proposal is to stop the fuzzers from generating BFGs when one side errors and the other doesn’t, or when the error messages don’t match, since this usually happens with queries that are already invalid. In this world, the fuzzer would only generate a BFG when both sides succeed, but return different results for the same query.

There is a risk that a new version of mongo could introduce a bug where a valid query suddenly errors. This is something the fuzzers would currently catch, but we should discuss how a problem like this would be identified if we were to make this change.