[SERVER-56044] [SBE] Fix various $changeStream tests timing out Created: 12/Apr/21  Updated: 06/Dec/22  Resolved: 18/Apr/21

Status: Closed
Project: Core Server
Component/s: Querying, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kyle Suarez Assignee: Backlog - Query Execution
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-55672 [SBE][replica_sets] postBatchResumeTo... Closed
Related
related to SERVER-55076 [SBE] assert.soon() timeout in jstest... Closed
related to SERVER-55010 Enable sharding suite against SBE bui... Closed
Assigned Teams:
Query Execution
Participants:

 Description   

Many tests in the SBE build variant that exercise sharding fail with

[js_test:lookup_change_stream_post_image_compound_shard_key] assert.soon failed: () => changeStream.hasNext() The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: f

A non-exhaustive list of these tests in jstests/sharding/:

  • jstests/sharding/change_stream_chunk_migration.js
  • jstests/sharding/change_stream_empty_apply_ops.js
  • jstests/sharding/change_stream_metadata_notifications.js
  • jstests/sharding/change_stream_transaction_sharded.js
  • jstests/sharding/change_streams_shards_start_in_sync.js
  • jstests/sharding/change_streams_unsharded_becomes_sharded.js
  • jstests/sharding/change_streams/lookup_change_stream_post_image_compound_shard_key.js
  • jstests/sharding/change_streams/resume_change_stream_on_subset_of_shards.js
  • jstests/sharding/change_streams/lookup_change_stream_post_image_hashed_shard_key.js
  • jstests/sharding/change_streams_delete_in_txn_produces_correct_doc_key.js
  • jstests/sharding/change_streams/lookup_change_stream_post_image_id_shard_key.js
  • jstests/sharding/change_stream_read_preference.js
  • jstests/sharding/change_stream_enforce_max_time_ms_on_mongos.js
  • jstests/sharding/change_streams_unsharded_update_resume.js
  • jstests/sharding/change_stream_update_lookup_collation.js
  • jstests/sharding/change_stream_error_label.js
  • jstests/sharding/change_streams_new_shard_new_database.js
  • jstests/sharding/change_stream_resume_from_different_mongos.js
  • jstests/sharding/change_streams/resume_change_stream.js
  • jstests/sharding/change_streams_update_lookup_shard_metadata_missing.js
  • jstests/sharding/change_stream_update_lookup_read_concern.js
  • jstests/sharding/change_streams/resume_change_stream_from_stale_mongos.js
  • jstests/sharding/change_streams_primary_shard_unaware.js
  • jstests/sharding/change_streams.js
  • jstests/sharding/change_streams_whole_db.js
  • jstests/sharding/change_stream_lookup_single_shard_cluster.js
  • jstests/sharding/change_stream_shard_failover.js

This ticket also encompasses assert.soon() failures related to change streams in the multiversion suite:

  • jstests/multiVersion/genericChangeStreams/change_streams_read_oplog_after_downgrade.js (task)
  • jstests/multiVersion/genericChangeStreams/change_streams_multi_version_cluster.js (task)

See also SERVER-55076



 Comments   
Comment by Githook User [ 19/Apr/21 ]

Author:

{'name': 'Drew Paroski', 'email': 'drew.paroski@mongodb.com', 'username': 'paroski'}

Message: SERVER-56044 Remove sbe_incompatible tag from change_streams_establishment_finds_new_shards.js
Branch: master
https://github.com/mongodb/mongo/commit/6d473a39a6ff33aa2ab945300d9a58b2113b0e28

Comment by Drew Paroski [ 18/Apr/21 ]

I verified that the patch that Arun is working on for SERVER-55672 will fix all of the tests in "jstests/sharding" listed above.

I also verified that Arun's patch will fix the multiversion failures listed above: https://spruce.mongodb.com/version/607a792f0305b920702f6d44/tasks

Closing this task as a duplicate of SERVER-55672.

Comment by Drew Paroski [ 17/Apr/21 ]

I confirmed by bisecting that the diff for SERVER-55498 also fixed whatever was causing the "jstests/sharding/change_streams_establishment_finds_new_shards.js" test to fail.

I've removed "jstests/sharding/change_streams_establishment_finds_new_shards.js" from the list above.

Comment by Kyle Suarez [ 12/Apr/21 ]

The error for sharding/change_streams.js doesn't look exactly like the others:

[js_test:change_streams] uncaught exception: Error: assert.soon failed, msg : expected to be able to see the first insert The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). :
[js_test:change_streams] doassert@src/mongo/shell/assert.js:20:14
[js_test:change_streams] assert.soon@src/mongo/shell/assert.js:382:17
[js_test:change_streams] runTest@jstests/sharding/change_streams.js:89:5
[js_test:change_streams] @jstests/sharding/change_streams.js:262:1
[js_test:change_streams] @jstests/sharding/change_streams.js:3:2
[js_test:change_streams] failed to load: jstests/sharding/change_streams.js

However that line in the test is just asserting we see the first change stream result in a timely manner, like the others:

    // Test that a change stream can see inserts on shard 0.
    assert.commandWorked(mongosColl.insert(makeShardKeyDocument(1000)));
    assert.commandWorked(mongosColl.insert(makeShardKeyDocument(-1000)));
 
    assert.soon(() => changeStream.hasNext(), "expected to be able to see the first insert"); // <---- fails

Comment by Kyle Suarez [ 12/Apr/21 ]

The error for js_test:change_stream_enforce_max_time_ms_on_mongos.js looks slightly different but I still believe this is because of a change stream not returning in a timely manner:

[js_test:change_stream_enforce_max_time_ms_on_mongos] uncaught exception: Error: 1800044 is not less than or eq 300000 :
[js_test:change_stream_enforce_max_time_ms_on_mongos] doassert@src/mongo/shell/assert.js:20:14
[js_test:change_stream_enforce_max_time_ms_on_mongos] _assertCompare@src/mongo/shell/assert.js:975:9
[js_test:change_stream_enforce_max_time_ms_on_mongos] assert.lte@src/mongo/shell/assert.js:991:9
[js_test:change_stream_enforce_max_time_ms_on_mongos] @jstests/sharding/change_stream_enforce_max_time_ms_on_mongos.js:156:1
[js_test:change_stream_enforce_max_time_ms_on_mongos] @jstests/sharding/change_stream_enforce_max_time_ms_on_mongos.js:7:2
[js_test:change_stream_enforce_max_time_ms_on_mongos] failed to load: jstests/sharding/change_stream_enforce_max_time_ms_on_mongos.js
[js_test:change_stream_enforce_max_time_ms_on_mongos] exiting with code -3

Generated at Thu Feb 08 05:38:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.