[SERVER-63781] $sortKey not filtered out in initial batch from post batch resume token Created: 17/Feb/22  Updated: 29/Oct/23  Resolved: 04/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.0-rc0, 4.4.18, 4.2.24, 5.0.13

Type: Bug Priority: Major - P3
Reporter: Jennifer Peshansky (Inactive) Assignee: Mickey Winters
Resolution: Fixed Votes: 0
Labels: pm-2419-commit
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
backported by SERVER-69002 [5.0] backport pm-2419 Per Shard Curs... Closed
backported by SERVER-69003 [4.4] backport pm-2419 Per Shard Curs... Closed
backported by SERVER-69004 [4.2] backport pm-2419 Per Shard Curs... Closed
Depends
is depended on by SERVER-63771 Fix all temporarily blocked tests in ... Closed
Duplicate
is duplicated by SERVER-63776 Unexpected Fields in per shard cursor... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: QE 2022-03-07
Participants:

 Description   

Failing test: shell_helper.js

Error: results != expected

Symptoms: sortKey not removed

 

variant file log run
ubuntu UBSAN AFF shell_helper.js logs 2
RHEL all feature flags metadata_notifications.js logs 3
RHEL apply_ops.js logs 2
RHEL AFF query patch apply_ops.js logs 2
ubuntu UBSAN AFF apply_ops.js logs 2
RHEL all feature flags apply_ops_resumability.js logs 2


 Comments   
Comment by Githook User [ 08/Nov/22 ]

Author:

{'name': 'Mickey. J Winters', 'email': 'mickey.winters@mongodb.com', 'username': 'mjrb'}

Message: SERVER-69004 backport per shard cursors

SERVER-62400
SERVER-62681
SERVER-62738
SERVER-63781
SERVER-63774
SERVER-63772
SERVER-63773
SERVER-58673
SERVER-70633
SERVER-69785

both modified: src/mongo/s/query/cluster_aggregate.cpp
both modified: src/mongo/s/query/cluster_aggregation_planner.cpp
both modified: src/mongo/s/query/cluster_aggregation_planner.h
these files had irreconcilable differences and the changes had to be reimplemented by hand.
this is due to some refactoring and AggregationTargeter not existing

(cherry-picked from commit e424f3ed1266bcd772c52bcf23c518b2fe6c83ab)
(cherry-picked from commit ba8f9a28c8769dfe10f73b190c943ee4a57ee7a1)
(cherry-picked from commit 1219ff764c932f16a68c7ba1afd9b925f9f876c8)
Branch: v4.2
https://github.com/mongodb/mongo/commit/792f237975470ab61e3e9cd4b54ede87ed9377d8

Comment by Githook User [ 31/Oct/22 ]

Author:

{'name': 'Mickey. J Winters', 'email': 'mickey.winters@mongodb.com', 'username': 'mjrb'}

Message: SERVER-69003 backport per shard cursors

SERVER-62400
SERVER-62681
SERVER-62738
SERVER-63781
SERVER-63774
SERVER-63772
SERVER-63773
SERVER-58673
SERVER-70633
SERVER-69785

(cherry-picked from commit 4d5451b20fecaf4a054cca94de6962fcf28c577f)
(cherry-picked from commit ba8f9a28c8769dfe10f73b190c943ee4a57ee7a1)
(cherry-picked from commit 1219ff764c932f16a68c7ba1afd9b925f9f876c8)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e424f3ed1266bcd772c52bcf23c518b2fe6c83ab

Comment by Githook User [ 07/Sep/22 ]

Author:

{'name': 'Mickey. J Winters', 'email': 'mickey.winters@mongodb.com', 'username': 'mjrb'}

Message: SERVER-69002 backport per shard cursors

SERVER-62400
SERVER-62681
SERVER-62738
SERVER-63781
SERVER-63774
SERVER-63772
SERVER-63773
SERVER-58673

(cherry-picked from commit fc54ebd0137a25ea664c022b51b685667dd037c7)
(cherry-picked from commit 53d7bceee61f73a1d6959edb5d490c3b338f3c0d)
(cherry-picked from commit 586663fec7c3a7d4a8b0185ff24825bd15e80dff)
(cherry-picked from commit ef2a62dcc27461d2be1b619c75bc04effa1f2021)
(cherry-picked from commit 4f3626ff4486e672569699dfde1cc0ae8c54d348)
(cherry-picked from commit 0f7683455bc06b153f14368a3f05f0b69671717e)
(cherry-picked from commit 11d01816f743d6764c4f12c42697f5edf813ce27)
(cherry-picked from commit 1fe77b5bd9fb13f9eb74275359dcc4ba69f2d5e9)
Branch: v5.0
https://github.com/mongodb/mongo/commit/4d5451b20fecaf4a054cca94de6962fcf28c577f

Comment by Githook User [ 03/Mar/22 ]

Author:

{'name': 'Mickey. J Winters', 'email': 'mickey.winters@mongodb.com', 'username': 'mjrb'}

Message: SERVER-63781 set per shard cursor initial batch size to 0
Branch: master
https://github.com/mongodb/mongo/commit/ef2a62dcc27461d2be1b619c75bc04effa1f2021

Comment by Jennifer Peshansky (Inactive) [ 28/Feb/22 ]

We need to override the initial batch size to be 0 in the initial aggregate command for a per shard cursor. This will establish the cursors. We can establish machinery using those cursors on the mongos side, and use that to get results. It will require another getMore to the shards. If this is an initial find with a tailable awaitData cursor, we return.

Comment by Mickey Winters [ 25/Feb/22 ]

the fix for this seems to just be only establishing the cursor when the initial aggregate command is sent so that there is no first batch. sharded change streams seems to have this behavior where if you have startAfter and there are events to be returned you have to call getMore to actually get them

Comment by Mickey Winters [ 25/Feb/22 ]

in shell_helpers.js only 1 event is returned and no getMore is issued. so the events with the $sortKey unfiltered seems to be returned with the first initial batch. this explains why for some of the failures when there is more than one change only the first change seems to have the unfiltered sort key

Generated at Thu Feb 08 05:58:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.