|
burn_in_tests.py failing when running the create_database.js FSM workload can be explained by how it doesn't define a $config.teardown function to clean up its unique database name. The intent for how FSM workloads are written after SERVER-30204 is implemented is that they shouldn't need to and would be discouraged from dropping any collections or databases at the end of the test because it'd undermine our ability to verify the data is consistent.
robert.guo, I hadn't thought too deeply about this while working on SERVER-30203 but we've lost coverage for running the data consistency checks before $config.teardown is called.
diff --git a/jstests/concurrency/fsm_libs/resmoke_runner.js b/jstests/concurrency/fsm_libs/resmoke_runner.js
|
index d94fd4e31c..bacd2070ae 100644
|
--- a/jstests/concurrency/fsm_libs/resmoke_runner.js
|
+++ b/jstests/concurrency/fsm_libs/resmoke_runner.js
|
@@ -19,6 +19,24 @@
|
function cleanupWorkload(workload, context, cluster, errors, header) {
|
const phase = 'before workload ' + workload + ' teardown';
|
|
+ try {
|
+ // Ensure that all data has replicated correctly to the secondaries before calling the
|
+ // workload's teardown method.
|
+ cluster.checkReplicationConsistency([], phase);
|
+ } catch (e) {
|
+ errors.push(new WorkloadFailure(
|
+ e.toString(), e.stack, 'main', header + ' checking consistency on secondaries'));
|
+ return false;
|
+ }
|
+
|
+ try {
|
+ cluster.validateAllCollections(phase);
|
+ } catch (e) {
|
+ errors.push(new WorkloadFailure(
|
+ e.toString(), e.stack, 'main', header + ' validating collections'));
|
+ return false;
|
+ }
|
+
|
try {
|
teardownWorkload(workload, context, cluster);
|
} catch (e) {
|
As far as adding the issue Vessy is running into, what do you think about having resmoke_runner.js call dropAllDatabases() immediately after setting up the cluster?
|
|
vesselina.ratcheva, could you provide a link to the patch build where you saw the burn_in_tests task fail? It should in general be possible to run FSM workloads with --repeat > 1. The concurrency framework drops the collection before starting to run the FSM workload as part of the prepareCollections() function.
I would also expect that FSM workloads which create their own collection or database (where the FSM workload name is used as a unique identifier) to currently drop those collections in their $config.teardown function. The intent is to address this in SERVER-30204 by having a version of the CleanEveryN that doesn't require restarting the entire MongoDB deployment after every FSM workload.
If there's something I'm missing and burn_in_tests.py cannot run FSM workloads multiple times, we should be able to add jstests/concurrency/fsm_workloads/*.js to the "exclude_files" section of burn_in_tests.yml.
|