[SERVER-39899] Enable the initial sync fuzzer in Evergreen Created: 01/Mar/19  Updated: 29/Oct/23  Resolved: 29/Mar/19

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.10

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Vlad Rachev (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Backwards Compatibility: Fully Compatible
Sprint: STM 2019-04-08
Participants:
Linked BF Score: 40
Story Points: 3

 Description   

Create a new initial_sync_fuzzer.yml resmoke.py YAML suite file that causes mongod processes to use an effectively infinite number of initial sync attempts.

buildscripts/resmokeconfig/suites/initial_sync_fuzzer.yml

test_kind: js_test
 
selector:
  roots:
  - jstestfuzz/out/*.js
 
executor:
  archive:
    tests: true
  config:
    shell_options:
      nodb: ''
      readMode: commands
      global_vars:
        TestData:
          # TODO: logComponentVerbosity?
          setParameters:
            numInitialSyncAttempts: 10000000

Define a new initial_sync_fuzzer_gen Evergreen task based on the existing rollback_fuzzer_gen Evergreen task.

## initial sync fuzzer ##
- <<: *jstestfuzz_template
  name: initial_sync_fuzzer_gen
  commands:
  - func: "generate fuzzer tasks"
    vars:
      <<: *jstestfuzz_config_vars
      # TODO: The number of files should be based on how the tests themselves take to run. We
      # should target a time for each generated task of ~10 minutes.
      num_files: ??
      num_tasks: 5
      npm_command: initsync-fuzzer
      resmoke_args: --suites=initial_sync_fuzzer
      name: initial_sync_fuzzer

Configure the new initial_sync_fuzzer_gen Evergreen task to run on all of the build variants the existing rollback_fuzzer_gen Evergreen task runs on with the exception of the "Enterprise RHEL 6.2 (inMemory)" and "Linux (ephemeralForTest)" build variants. Since the initial version of the initial sync fuzzer is meant to only be targeting the interaction between initial sync and prepared transactions, we can only run it against the WiredTiger storage engine.

  • Enterprise RHEL 6.2
  • Enterprise RHEL 6.2 (majority read concern off)
  • Windows 2008R2 DEBUG
  • macOS
  • Enterprise RHEL 6.2 DEBUG Code Coverage
  • ASAN Enterprise SSL Ubuntu 16.04 DEBUG
  • UBSAN Enterprise Ubuntu 16.04 DEBUG


 Comments   
Comment by Githook User [ 29/Mar/19 ]

Author:

{'name': 'vrachev', 'username': 'vrachev', 'email': 'vlad.rachev@mongodb.com'}

Message: SERVER-39899 Enable the initial sync fuzzer in Evergreen
Branch: master
https://github.com/mongodb/mongo/commit/13af8cb458bd09268c91aa5133ca1725a498c01a

Comment by Vlad Rachev (Inactive) [ 28/Mar/19 ]

Thanks max.hirschhorn. I should've run the other initial sync tests before pushing. 

Comment by Max Hirschhorn [ 28/Mar/19 ]

vlad.rachev, I reverted the changes from 6b8b02e because they were causing the jstests/replsets/initial_sync_test_fixture_test.js test to fail. It looks like the restartNodeWithoutData() function needs to check whether TestData.logComponentVerbosity is defined before attempting to pass it as a server parameter to the server.

[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.410+0000 2019-03-27T21:46:44.410+0000 I  -        [js] shell: started program (sh76294):  /data/mci/dd5874ef7507b5325dbd98f2dd0d8693/src/mongod --oplogSize 40 --keyFile jstests/libs/authTestsKey --port 21771 --replSet InitialSyncTest --dbpath /data/db/job7/mongorunner/InitialSyncTest-1 --setParameter failpoint.initialSyncFuzzerSynchronizationPoint1={ "mode" : "alwaysOn" } --setParameter logComponentVerbosity=undefined --setParameter writePeriodicNoops=false --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --setParameter enableTestCommands=1 --setParameter enableLocalhostAuthBypass=false --setParameter disableLogicalSessionCacheRefresh=true --storageEngine wiredTiger --setParameter transactionLifetimeLimitSeconds=86400 --setParameter orphanCleanupDelaySecs=1 --enableMajorityReadConcern true --wiredTigerCacheSizeGB 1
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.430+0000 d21771| BadValue: Bad value for parameter "logComponentVerbosity": code FailedToParse: FailedToParse: Expecting '{': offset:0 of:undefined
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.430+0000 d21771| try '/data/mci/dd5874ef7507b5325dbd98f2dd0d8693/src/mongod --help' for more information
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.611+0000 Could not start mongo program at 21771, process ended with exit code: 2
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.611+0000 2019-03-27T21:46:44.611+0000 E  QUERY    [js] uncaught exception: Error: Failed to start node 1 :
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 ReplSetTest/this.start@src/mongo/shell/replsettest.js:2343:19
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 restartNodeWithoutData@jstests/replsets/libs/initial_sync_test.js:147:21
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 InitialSyncTest/this.step@jstests/replsets/libs/initial_sync_test.js:223:13
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 @jstests/replsets/initial_sync_test_fixture_test.js:78:13
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 @jstests/replsets/initial_sync_test_fixture_test.js:15:2
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 2019-03-27T21:46:44.612+0000 F  -        [main] failed to load: jstests/replsets/initial_sync_test_fixture_test.js
[js_test:initial_sync_test_fixture_test] 2019-03-27T21:46:44.612+0000 2019-03-27T21:46:44.612+0000 E  -        [main] exiting with code -3

https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_display_replica_sets_auth_6b8b02e8d72c055c6f686a781e53cde9384461b3_19_03_27_20_01_38

Comment by Githook User [ 28/Mar/19 ]

Author:

{'email': 'max.hirschhorn@mongodb.com', 'name': 'Max Hirschhorn', 'username': 'visemet'}

Message: Revert "SERVER-39899 Enable the initial sync fuzzer in Evergreen"

This reverts commit 6b8b02e8d72c055c6f686a781e53cde9384461b3.
Branch: master
https://github.com/mongodb/mongo/commit/a1c5787c45462a2eb190e09eab44d0b62a843b76

Comment by Githook User [ 27/Mar/19 ]

Author:

{'email': 'vlad.rachev@mongodb.com', 'name': 'vrachev', 'username': 'vrachev'}

Message: SERVER-39899 Enable the initial sync fuzzer in Evergreen
Branch: master
https://github.com/mongodb/mongo/commit/6b8b02e8d72c055c6f686a781e53cde9384461b3

Comment by Judah Schvimer [ 05/Mar/19 ]

I would just put replication.initialSync at level 4 since that should cover everything we have or will add. It's a very limited set of log messages, and if they're useful for anything this is what they should be useful for.

Comment by Max Hirschhorn [ 04/Mar/19 ]

I would start at either 0 or 2. I don't think 1 adds much for initial sync. If we wanted to log every op we apply or every document we insert we could see the least verbose way to do that.

Initial sync has it's own log component so logging just that at a very high level, and everything else at a much lower level definitely makes sense.

judah.schvimer, are the LOG(3) message in databases_cloner.cpp something you're likely interested in? initial_syncer.cpp logs information about timestamping at "replication.initialSync"=2 that was was curious if that'd be sufficient.

Comment by Judah Schvimer [ 04/Mar/19 ]

Initial sync has it's own log component so logging just that at a very high level, and everything else at a much lower level definitely makes sense.

Comment by Judah Schvimer [ 04/Mar/19 ]

I would start at either 0 or 2. I don't think 1 adds much for initial sync. If we wanted to log every op we apply or every document we insert we could see the least verbose way to do that.

Comment by Max Hirschhorn [ 01/Mar/19 ]

judah.schvimer, do you have any recommendations for what logComponentVerbosity we set for the initial sync fuzzer?

Generated at Thu Feb 08 04:53:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.