[SERVER-78485] $project after $search adds 20x latency Created: 27/Jun/23  Updated: 14/Sep/23

Status: In Progress
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Evan Darke Assignee: Will Buerger
Resolution: Unresolved Votes: 0
Labels: greenerbuild, quick-tech-debt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File no_project_stats.js     File project_stats.js    
Assigned Teams:
Query Integration
Operating System: ALL
Sprint: QI 2023-07-24, QI 2023-08-07
Participants:

 Description   

 

 

 

 

Materializing the first batch without limit takes 736ms 
measure(() => db.listings.aggregate([{ $search: { queryString: { defaultPath: "data.host_type", query: "h*" }} }])) 

 

Adding a $project stage should not add significant overhead but:

 

Materializing the first batch with limit takes 13,407ms
measure(() => db.listings.aggregate([{ $search: { queryString: { defaultPath: "data.host_type", query: "h*" }} }, { $project: { highlight: { "$meta": "searchScore" } }}])) 

 

$project itself isn't slow though because the same query with a really large limit is very fast

Materializing the first batch takes 156ms
measure(() => db.listings.aggregate([{ $search: { queryString: { defaultPath: "data.host_type", query: "h*" }} }, { $project: { highlight: { "$meta": "searchScore" } }}, {$limit: 1000} ])) 

 

 

 

Use cases:

  • We have reports of users experiencing timeouts when iterating over the entire resultset of a $search for batch processing. They address this by issuing many smaller queries using skip+limit, but this is also slow for large skip values.
  • Developers debugging on commandline often forgo using limits, they shouldn't see 13s latency overhead for 


 Comments   
Comment by Evan Darke [ 25/Jul/23 ]

kyle.suarez@mongodb.com Unfortunately I think I've only tried this on sharded clusters running 6.0

Generated at Thu Feb 08 06:38:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.